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(54) Title: A METHOD FOR THE AUTOMATIC COMPUTERIZED AUDIO VISUAL DUBBING OF MOVIES 
(57) Abstract 

A method using computer software for automatic au- 
dio visual dubbing (5), using an efficient computerized auto- 
matic method for audio visual dubbing of movies by comput- 
erized image copying of the characteristic features of the lip 
movements of the dubber onto the mouth area of the original 
speaker. The invention uses a method of vicinity-searching, 
three-dimensional head modeling of the original speaker (3), 
and texture mapping (10) technique to produce new images 
which correspond to the dubbed sound track. The invention 
thus overcomes the well known disadvantage of the conela- 
tion problems between lip movement in an original movie and 
the sound track of the dubbed movie. 
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A METHOD FOR THE AUTOMATIC COMPUTERIZED AUDIO VISUAL 

DUBBING OF MOVIES 

FIELD OF THE INVEHTION 

The present invention relates to a method for automatic 
audio visual dubbing. More specifically the 

said invention relates to an efficient computerized 
automatic method for audio visual dubbing of movies by 
computerized image copying of the characteristic features 
of the lips movements of the dubber onto the mouth area 
of the original speaker • The present invention uses a 
method of vicinity-searching , three-dimensional head 
modeling of the original speaker, and tescture mapping 
techniques in order to produce the new images which 
correspond to the dubbed sound track. 

The invention overcomes the well known disadvantage of 
the correlation problems between lips movement in the 
original movie and the sound track of the diibbed movie. 

DEFINITIONS OF TERMS RELATED TO TEE INVENTION 

First are provided some definitions of important key 
words employed in this specification. 

Actor (the original actor) - an actor, speaker, singer, 
animated character, animal, an object in a movie, or a 
subject in a still photograph- 
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Audio Visual Dubbing - Manipulating, in one or more 
frames, the mouth area of the actor so that its status 
will be similar as much as possible to that of the dubber 
in the reference frame. 

Correlation Function - A function describing the 
similarity of two image regions. The higher the 
correlation, the better is the match. 

Dubber - The person or persons, who 
speak/narrate/sing/interpret the target text. The dubber 
can be the same as the actor. 

Dubbing - Replacing part or all of one or more of the 
original sound tracks of a movie, with its original text 
or sounds (including the case of the silent track of a 
still photograph), by another sound track containing the 
target text and/or soimd. 

Edge Detector - A known image processing technigue used 
to extract boundaries between image regions which differ 
in intensity and/or color. 



Face Parametrization - A method that numerically 
describes the structure, location, and expression of the 
face. 
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Head Model - A three-dimensional wire frame model of the 
face that is controlled by numerous parameters that 
describe the exact expression produced by the model (i.e. 
smile, mouth width, jaw opening, etc.)* 

Movie (the original movie) - Any motion picture (e.g. 
cinematic feature film, advertisement, video, animated 
cartoon, still video picture, etc.)- A sequence of 
consecutive pictures (also called frames) photographed in 
succession by a camera or created by an animator. In the 
case of the movie being a still photograph, all of the 
consecutive pictures are identical to each other. When 
shown in rapid succession an illusion of natural motion 
is obtained, except for the case of still pictures. A 
sound-track is associated with most movies, which 
contains speech, music, and/or sounds, and which is 
synchronized with the pictures, and in particular where 
the speech is synchronized with the lip movements of the 
actors in the pictures. Movies are realized in several 
techniques. Common methods are: (a) recording on film, 
(b) recording in analog electronic form ("video"), (c) 
recording in digital electronic form, (d) recording on 
chips, magnetic tape, magnetic disks, or optical disks, 
and (e) read/write by magnetic and/or optical laser 
devices. Finally, in our context, an "original movie" is 
also an audio-visual movie altered by the present 
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invention, which serves as a base for further 
alterations . 

Original Text - A text spoken or sung by the actor when 
the movie is being made, and which is recorded on its 
sound track. The text nay be narrated in the background 
without showing the speaker, or by showing a still 
photograph of the speaker. 

Pixel - Picture element. A digital picture is composed of 
an array of points, called pixels. Each pixel encodes the 
numerical values of the intensity and of the color at the 
corresponding picture point. 

Reference Similarity Frame - a picture (being a frame 
in the original movie, a frame in any other movie, or a 
still photograph) in which the original actor has the 
desired features of the mouth-shape and head posture 
suitable for the audio visually dubbed movie. 

Target Text - A new vocal text, to replace the original 
vocal text of the actor. The target text may also be that 
which is to be assigned to an actor who was silent in the 
original movie. The new text can be in another language, 
to which one refers as DUBBING. However, this invention 
relates also to replacement of text without changing the 
language, with the original actor or with a dubber in 
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that same language. The target text may have the same 
meaning as the original text, but may have also a 
modified, opposite, or completely different meaning. 
According to one of many applications of the present 
invention, the latter is employed for creation of new 
movies with the same actor, without his/her/its active 
participation. Also included is new vocal text used to 
replace the null vocal text attached to one or more still 
photographs . 

Texture Mapping - A well known technique in computer 
graphics which maps texture onto a three-dimensional wire 
frame model. 

Two-Dimensional Projection - The result of the rendering 
of the three-dimensional face model onto a two- 
dimensional device like a monitor, a screen, or 
photographic film. 

BACKGROUND OF THE INVENTION 

Movies are often played to an audience that is not 
familiar with the original language, and thus cannot 
understand the sound track of such movies. Two well known 
common approaches exist to solve this problem, in one 
approach sub-titles in typed text of the desried language 
are added to the pictures, and the viewers are expected 
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to hear the text in a foreign language and simultaneously 
to read its translation on the picture itself, such 
reading distracts the viewers from the pictures and from 
the movie in general. Another approach is dubbing, where 
the original sound-track with the original text is being 
replaced by another sound-track with the desired 
language. In this case there is a disturbing mis-match 
between the sound-track and the movements of the mouth. 

There have been some earlier attempts to overcome these 
disadvantages, none of which have been commercialized 
because of inherent principal difficulties which made the 
practical execution unrealistic. Thus, in U.S. Patent 
Number 4,500,281 a method is described which performs the 
measurements of the shape of the mouth manually by a 
ruler or with a cursor, and corrects the mouth shape by 
moving pixels within each frame. As will be seen in the 
description of the invention, the method according to the 
present invention is inherently different and much 
superior in the following points: m the present 
invention the tracking of the shape of the mouth is done 
automatically and not manually, m the present invention 
changing the shape of the mouth is done by using a three- 
dimensional head model, for example like those described 
by P.Ekraan and W.V.Friesen, (Manual for the Facial Action 
Unit system, Consulting Psychologist Press, Palo Alto 
1977). In the present invention the mouth area of the 
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actor is replaced using the mouth area of a reference 
similarity frame, in the present invention mouth status 
parameters of the dubber are substituted for mouth status 
parameters of the actor. 

The U.S. Patent Number 4,260,229 relates to a method of 
graphically creating lip images. This U.S. patent is 
totally different from the present invention: In the U.S. 
Patent, speech sounds are analyzed and digitally encoded. 
In the present invention no sound analysis is done; nor 
is any required at all. 

To make for better viewing of the audio visually dubbed 
movie, the present invention provides a computerized 
method wherein, in addition to replacing the sound track 
to the target text> the mouth movements of the actor are 
being automatically changed to match the target text. The 
new mouth movements are linguistly accurate and visually 
natural looking according to all of the observable 
parameters of the actor's face. 

SUMMARY OF THE INVENTION 

The present invention provides a method for automated 
computerized audio visual dubbing of movies, comprising 
of the following steps (see Pig. i): 
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(a) selecting from the movie a frame having a picture, 
preferably frontal, of the actor's head and, if 
available, a frame with its side profile; 

(b) marking on the face several significant feature 
points and measuring their locations in the frame; 

(c) fitting a generic three-dimensional head model to the 
actor's two-dimensional head picture by adapting the data 
of the significant feature points, as measured in stage 
(b), to their location in the model; 

(d) tracking of the said fitted three-dimensional head 
model parameters throughout the movie, from one frame to 
its successor, iteratively in an automated computerized 
way and creating a library of reference similarity 
frames . 



(e) taking a movie of a dubber wherein the dubber speaks 
the target text; 



(f) repeating stages (a), (b) , (c) , and (d) with the 
dubber; 

(g) normalizing the dubber 's minimum and maximun. values 
Of each parameter to the actor's minimum and maximum 
valuse of the same parameters; 
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(h) mapping, on a frame to frame basis, the two- 
dimensional actors face onto its three-dimensional head 
model by using a texture mapping technique, making use of 
reference similarity frames; 

(i) changing the texture mapped three-dimensional model 
obtained in stage (h) by replacing, on a frame to frame 
basis, the original mouth parameters with the mouth 
parameters as contputed in stage (d) for the dubber and 
obtaining the parametric description for the new picture, 
with identical values to the original, except that the 
actor's mouth status resembles the mouth status of the 
dubber; 

(j) texture mapping the lips area of the same actor from 
a frame in the movie, with identical or very similar 
mouth status to the desired new mouth status, onto the 
lips area of the actor's head model for the current frame 
and then projecting the lips area from the actor's head 
model onto the current new frame. (This stage is 
optional , depending on the application . ) 

By using the three dimensional head model one can control 
the audio visual dubbing process even if the actor is 
moving his head. In most applications about 15 
significant feature points on the face are used in the 
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tracking stage, such as eye corners, mouth corners, and 
the nostrills. only those feature points which are 
visible to the viewer (using the information available to 
the model) are tracked. 



In the present invention, audio visual dubbing is 
normally used in conjunction with the use of audio 
dubbing; but one may also use it in conjunction with an 
audio track where no equivalent track exists in the 
original movie. 

The method according to the present invention is useful 
for the audio visual dubbing of motion pictures such as 
cinematic feature films, advertisments , video, and 
animated cartoon. Also the audio visual dubbing of still 
photographs, wherein all of the frames of the movie are 
the same, is made possible by the present invention. For 
instance, still photographs are used for this type of 
movie in T.V. news programs where the reporter's voice is 
heard while a still photograph of him/her is shown. 

Thus, according to the present invention even speechless 
actors, infants, animals, and inanimate objects can be 
audio visually dubbed to speak in any language. 

According to our invention, the animation process saves 
much of the labor associated with the animation of the 
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mouth area. 

The present invention further provides a computer program 
(see Appendix 1) for operating the computerized audio 
visual dubbing. 

The present invention further relates to the library of 
reference similarity frames as created in Step d (above). 

DETAILED DESCRIPTION OF THE INVENTION 

Given an original movie where an actor speaks the 
original text, a movie of a dubber is made where the 
dubber speaks the target text in either another 
language or the same language. The movie of the dubber 
is taken while the dubber performs a routine dubbing 
adaptation of the original into the target text. 

This invention provides a method for the changing of the 
actor's facial motions in the original movie to create a 
new movie having the sound-track in the target text from 
the movie of the dubber, while the pictures are of the 
original movie, whereas the motion of the mouth of the 
actor is modified to correspond to the new sound-track. 

For brevity considerations, the description of this 
invention uses pictures in electronic digital form 
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(composed of an array of pixels), but movies in any other 
form are treatable as well. In these cases, the movie is 
translated to a digital form by existing techniques, 
manipulated in the digital form, and returned back to any 
desired form by known techniques. 

A facial expression can be described by an "action unit", 
for example the Facial Action Coding System (FAGS) by 
Ekman and Friesen (Ekman et. al.)- Action units (AU) 
stands for a small change in the facial expression which 
depends on a conscious activation of muscles (H. Li, P. 
Roivainen, Forchheimer, 3-D Motion in Model-Based 
Facial Image Coding, IEEE Transactions in PAMI, 15 (2), 
545—555 (1993)). The AU information is expressed in 
parameter form. Using the AU parameters, many facial 
expressions can be controlled. Parameters like face 
location and size, aspect-ratios of face regions, 
location of specific face fetures and many more. 

As explained above, one of the stages of this invention 
is a three-dimensional parameterisation of the face. An 
example for one such model is the model of Parke 
(Fredric I. Parke, Parameterized Models for Fatial 
Animation, IEEE computer Graphics and Applications, 12 
(11), 61-68, (1982)) which consists of about 25 
parameters. Face parameters can be roughly divided into 
three main classes: structure parameters, location 
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parameters, and es^pression parameters. 

structure parameters are fixed for every head and include 
distance ratios between the mouth and the eyes, the mouth 
and the chin, width of the model, jaw width, etc. The 
location parameters are, for example: three parameters 
for three-dimensional rotation in space and three 
parameters for three-dimensional translation (position in 
the real world). The expression parameters are, for 
instance: mouth width, smile (as an example, the 
parameter values here may be o.o for a very sad mouth and 
1.0 for a very happy mouth), jaw opening, upper lip 
lifting, lower lip lowering, lip thickness, and so on. 

Using a face model, the present invention is centered on 
a computer program (see Appendix l), which automaticly 
re-shapes the lip movements of the actor according to the 
lip movements of the dubber by searching for the nearest 
reference similarity frames. This computer program 
(software), or similar, is an integral and an important 
part of the present invention. The process, according to 
the present invention, is divided generally into the 
TRACKING phase and the NEW-MOVIE generation phase, as 
follows : 

I. TRACKING phase 

Step 1: The first step is personalizing the generic 
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three-dimensional face model for both the actor and the 
dubber. In order to modify the generic face model to fit 
a specific face, some additional information is needed. 
The generic model has to be translated, scaled and 
stretched to fit the given actor's face, from its initial 
position and setting. This Is done by manually pointing 
using a pointing device such as a mouth, a touch screen, 
etc., several facial feature points on the actor's face, 
e»g, eye corners, mouth corners, top and bottom of face. 
Typically a total of approximately 15 feature points are 
used, but this number may vary according to 
specifications. These feature points are marked on one 
(any) of the frames in the movie, in which the actor, 
preferably, faces the camera. The computer program then 
calculates automatically the exact model parameter's 
modifications needed for its two-dimensional projection 
to fit the actor face on the movie frame. In addition to 
using the facial feature points and in order to increase 
accuracy, the model is also adjusted to match the head 
edges, which are computed using an edge detector. If a 
side view of the actor is available it can be used to set 
several depth parameters, such as face depth and nose 
length. Otherwise, the face depth is scaled by some 
predetermined scale which is set experimentally. 



Step 2: After the generic face model has been 
personalized to the desired actor, face features in 
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several key frames in the movie are marked. The number of 
such frames can vary from a single first frame to about 
5% of all frames, depending on the difficulty of. the 
segment fitting the model to the actor using the marked 
facial features in those key frames achieves a 
stabilization of the automatic tracking (described 
later), and these key frames assures stable and 
continuous tracking. Neact, the program calibrates 
according to several examples of mouth shapes, later to 
be used for mouth tracking. Finally, the range of the 
mouth parameters (mimimum and maximum values) for the 
specific actor are estimated using all the values of the 
model parameters fitted to all key frames. 

Step 3: The next stage is the automatic tracking of the 
actor •s face throughout the entire movie: This is 
performed from one frame to its successive frame, using 
the face model, in two steps: first, the two-dimensional 
face of the actor is mapped onto the three-dimensional 
face model using a texture-mapping technique. The model 
can now be altered by changing its parameters only, 
creating new, synthetic images, which are otherwise very 
similar to the original movie frames: everything remains 
unchanged except for different face location, its 
orientation, and its expressions* By using a minimization 
algorithm, either analytic or numerical (such as steepest 
descent algorithm) , the program now computes those 
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parameters which, naxiadze the correlation function 
between the face area of the actor in the next frame and 
the synthesized projection of the texture-mapped face 
model. The steepest descent algorithm increases or 
decreases parameters in the direction that increases the 
correlation function. It can either work for each 
parameter s^arately (until it maximizes the 
correlation), or it can modify all the parameters at 
once. 

Step 4: After the model is locked on the head of the 
actor in the next frame, the mouth has to be tracked. 
This is done by first, checking the parameters of all of 
the mouths in the key frames and in several previous 
frames already tracked. Then, the frame that gives the 
higher correlation is choosen as a first guess for the 
tracking. Next, the same minimization algorithm used to 
track the global head motion is used, until the 
correlation function has maximized . The parameters 
describing the face model in the tracked frame are 
written into a file for later use. 

Step 5: Steps 3 and 4 are repeated until the entire movie 
is processed. For best results, instead of executing this 
process serially from the first frame to the last, the 
key frames can be used as initial points of tracking. 
Every two consecutive key frames are used to track from 
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each of them to the frames between them. That way, 
stabilasation of the tracking is preserved. 

Step 6: The tracking described above is applied to the 
dubber movie as well. 

II. NEW MOVIE generation phase 

This phase combines the tracking results of both the 
original and the dubber 's movies, in order to synthesize 
the new audio visually dubbed movie. Biis audio visually 
dubbed movie, as explained above, is mostly formed out of 
the original movie, except for the face of the actor in 
this audio visually dubbed movie. This face is a texture- 
mapped face on the three-dimensional face model, 
synthesized, as described above, to fit the lip, mouth, 
and cheek shapes of the dubber at that particular time. 
Thus, the parameters of the face model computed as 
described in phase I, is used to produce the new audio 
visually dubbed movie, in which for every frame in the 
original movie, the mouth parameters are modified to 
those of the dubber. The exact process is as follows: 

Step 7: For every frame in the original movie, the face 
of the actor is texture-mapped on the appropriate face 
model using the parameters that were calculated in step 3 
for the original movie. The mouth parameters of the 
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dubber as calculated in step 3 are used as follows for 
the new audio visually dubbed movie. 

Step 8: Once the desired mouth-shape of the actor is 
known, the original movie is being searched in the 
neighborhood of the current frame (approximately o.i - lo 
seconds forwards and backwards in time) for a mouth that 
is most similar in shape or parameters to the new desired 
mouth. This search for the reference similarity frame 
takes into account the mouth-shape already chosen for the 
previous frame in order to make the mouth motion smooth 
and continuous. From the several (5 - lo) best fit 
mouths, the mouth which is picked is from the frame that 
is closest in time to the previous picked mouth. 

Step 9: The mouth chosen in step 8 is texture -mapped into 
the mouth model using its pre-computed parameters. The 
face model parameters are then changed to the desired 
mouth shape, producing a very realistic new frame, which 
replaces the old frame in the original movie. The user of 
the program can choose the desired mouth area to be 
texture-mapped in place - it can" be either the inside of 
the mouth, the whole mouth including the lips or even a 
bigger area. This procedure creates a synthesized image, 
in which the face around the mouth, and in particular the 
lips, are re-shaped according to the sound track, while 
retaining the familiar face of the original actor. Step 
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8 can also be skipped, so that the inside of the mouth 
will be empty. This is useful for making a talking movie 
from still picture, where the inside information of the 
mouth is missing; because in the reference frame 
similarity dictionary of the actor there does not exist 
any near fit of lip shapes for the target. This black 
interior can also be filled with visual color /texture . 

Step 10: Finally, the sound track from the dubbed movie 
(target text) replaces the original text and sound. 

It is to be noted that an animator using the invented 
software tool, is free to modify, set, or fix any of the 
head or mouth parameters, in both the original or audio 
visually dubbed movie, and even pick a specific mouth to 
be textured-mapped in place, as described in step NM8, 
all of these at any of the above stages. The tracking 
program is highly interactive and user-friendly. 

The involved software (see Appendix 1) of this invention 
is very versatile, and can be used in a very wide array 
of sound/text replacement applications many of which have 
been mentioned above. The following are examples of some 
applications of the present invention: 

Advertising: For products sold world-wide, an original 
advertising commercial can be manipulated to produce the 
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sane commercial in any desired language. This saves the 
need to produce a new video for every country or language 
that the product is aimed for. 



Another possibility is to edit a movie, by altering 
existing scenes without having to re-shoot them again. 
If, for example, after the movie production was over, the 
director/editor wishes to alter a specific scene, or 
change one sentence of a specific actor. 

The present invention refers not only to narrated text 
but also to songe, operas, and music, opening the 
possibility to change the language of musical video 
clips . 



The production of an animated cartoon is assisted by 
drawing a line segment for the actor's mouth, drawing a 
small actor's picture dictionary containing representive 
reference similarity frames with completely drawn 
mouths, and then allowing these lip-line segments to be 
replaced with the coresponding lip shapes of the dubber 
as are to be found in the actor's picture dictionary. 

In general, applications of a method for automatic audio 
visual dubbing of movie include: cinematic movies, 
cartoons, documentaries, advertizing, news, educational 
programs, court documentations, speaches, lectures. 
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historical documentation, hearing coimaittees , home 
videos, sports events, entertainment events, operas, 
musicals, musical video-clips, simultaneous translation, 
and adding speach to sequences of either original or 
added still frames of the aforesaid. 

Furthermore, using the library of the previously 
described reference similarity frames, the present 
invention allows one to create new movies altogether, and 
also to convert background narrative to audio visual 
speach, and to audio-visualize any written text. 

The present invention will be further described by 
Figures 1-4. These figures are solely intended to 
illustrate the prefered embodyment of the invention and 
are not intended to limit the scope of the invention in 
any manner. Likewise the attached software (Appendix i) 
represents an example of the implementation of the method 
disclosed in the present patent and is not intended to 
limit the scope of said method in any way. 

Figure l illustrates a block diagram showing the method 
stages, and its contents is is detailed below. 

Figures 2a and 2b illustrate an example of significant 
points on a generic frontal picture of a head (Figure 2a) 
and a generic side profile picture of a head (Figure 2b). 
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Figure 3 shows an example of a generic wire frame face 
model . 

For illustrative purposes one can take the significant 
points shown in Figures 2, measure them on pictures of a 
real actor, and apply them to a generic wire frame face 
model (Figure 3). Fitting a three-dimensional head model 
to the actor's two-«dimensional head picture by adapting 
the data of the significant points, as measured, results 
in an integration, as can be seen in Figures 4a and 4b. 

Figure 4a is an example showing how a custom fitted wire 
frame model fits onto a frontal view of an actor's face. 

Figure 4b is an example showing how a custom fitted wire 
frame model fits onto a profile view of an actor's face. 

Figure l illustrates a block diagram showing the method 
stages : 

In the original movie (1) a frame is selected (2) having 
a nearly frontal picture of the original actor's head 
and, if available, a frame is also selected having a 
picture with his side profile. 

A three-dimensional head model is fitted to the actor's 



wo 97/15926 



23 



PCTAB96/01056 



two dimensional head picture(s). This model can be 
controlled by several parameters such as for the 
position of the head and the status of the mouth. This 
fitting stage (3) is done by adapting the data of the 
significant points, as measured in the selected frame, to 
the model. 

The next step (4) is the automated computerized tracking 
of the fitted three-dimensional head model parameters 
throughout the movie, from one frame to the next, one 
partial or complete three-dimensional head model for each 
frame where the actor appears, is used. Any of the 
resulting frames can serve as a reference similarity 
frame for the lips replacement. 

A movie of the dubber is taken (5). In this movie the 
dubber faces the camera in most of the frames. The dubber 
speaks the target text in this movie. 

The same process as applied to the original actor's movie 
is applied to the dubber 's movie: A frame form the 
dubber "s movie is selected (6) having a frontal picture 
of the dubber 's head, and if available, a frame with a 
picture of his side profile. A three dimensional head 
model is fitted to the dubber 's two dimensional head 
picture (7) by adapting the data of the significant 
points, as measured in the selected frame, to the 
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dubber*s model. An automated computerized tracking (B) of 
the said dubber's fitted three dimensional head model 
parameters is taken throughout the movie, from one frame 
to the next. 

The next stage in this method is to normalize (9) the 
dubber's minimum and maximum parameters to the actor's 
minimum and maximum parameters. 

In a frame to frame fashion, the original actor's two 
dimensional face is mapped (lo) onto his three 
dimensional head model. This mapping stage is done by 
using a texture mapping technique with the reference 
similarity frames. The result of this stage is one mapped 
three dimensional partial head model of the original 
actor for each frame of the original actor in, the 
original movie; wherein the model, corresponding to a 
given frame may be complete in the case when the original 
frame contains a frontal view of the actor's face. 

In the next stage the textured three dimensional model 
frames obtained for the original actor are changed (ii) 
by replacing, on a frame to frame basis, the original 
mouth parameters with mouth parameters as computed for 
the dubber in the corresponding frames; correspondance 
being determined by the desired sound track substitution 
(i.e. the dubbing). Thus is obtained the parametric 
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description for a new picture identical to the original, 
except that the actor's mouth status resembles the mouth 
status of the dubber; wherein the new picture coresponds 
to a frame in the new audio visually dubbed movie. 

In order to overcome difficulties like for example those 
that arise when the dubber in (8) opens a closed mouth 
in (4), a frame or frames in the original movie are 
sought whose mouth status is similar to the desired new 
mouth status. These frames, termed reference similarity 
frames, are usually but not necessarily in a temporal 
proximity to the processed frame, and the lips from that 
frame are copied using texture mapping (12) into the lips 
area in the new frame. The search for a reference 
similarity frame is an essential component of the present 
invention. We therefore repeat its definition: a 
reference similarity frame is a picture (being a frame 
in the original movie, a frame in any other movie, or a 
still photograph) in which the original actor has the 
desired features of the mouth-shape and head posture 
suitable for the audio visually dubbed movie. 

Alternatively, the reference similarity frame may be 
taken from a compiled picture library of the original 
actor or of other actors. 
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The process (12) is repeated all over again for each 
frame, until the entire movie is converted. 
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FACE IMAGING Ltd source code for program 'dub' 

The following printout includes the source for the 
following modules: 



dub. c 
general ,c 
gimg. h 

mv.h 
list.h 

io.c 

texture .c 



main program 

general functions and utilities 
interface for gimg.c - image I/O and 
filtering 

interface for mv.c - moving handling 

interface for list.c - list data type 

handl ing 

I/O functions 

texture mapping stuff 



These modules comprises the program 'dub' which 

automatically fixes the lips of a person in a movie to 

another person's lip shapes, as described in the 
invention description. 



The modules included are the main modules. Utility 
modules, such as handling images, movies, etc- are not 
included. 



SUBSTITUTE SHEET (RULE 26) 



wo 97/15926 



28 



PCT/IB96/01056 



* Module: dub.c 

♦ By: Ranco 

♦ History: 

* 18.8.94: creation. 

^include <evciit.h> 

^include "x^inch" 

#include "defs.h" 

#include "image.h** 

^include "general.h" 

^include "mv.h" 

#inciude "vars.h" 

#include "^dub-h" 

#includc "gimg.h" 

^include "io.h" 

^include "tcxturc.h" 

#include <math.h> 

^include <dmedia/cl_cosmo.h> 

#iiiclude <audiofile.h> 

Movie niovie,out_inoviel, out_movic2,trans_inovic; 
XtAppContext ac; 
Widget tl, da; 
Display ♦dpy; 
GC gc; 

char output_movie_namc[MAX_NAME]. scurce^movie^namefMAX^NAME], 
trans^movie_name[MAX_NAME],dubbed_inovie_name[MAX_NAME], 
force_fi^es_filciianie[MAX__NAME].fiill_dubbed_movie_name[MAX_NAME^ 
audio_file_,name[MAX_N AME] ; 

int first^frame, last_fTaine, translator_first_frame; 

int sample = l,frame_range = FRAME_AREA, jpeg_quality; 

float mouth_scale; 

int avgjjarams_flag = FALSE, avg^width^flag = TRUE, forcc_franics_flag = FALSE; 
int fulLflag = FALSE; 

Parameter **!efl_sourcc_aiTay, **righl_source_array; 
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Parameter **left_trans_aiTay, •*right_trans_aiTay; 
extern int tcxturc^map; 
Gimg curr^framcl; 

int one_frame_niode, normahze^mouth, verbose^mode; 

int which_params[] 
{JAW_R0TAT10N,Krra_Yscl,RSE_UPLIP,LWIU-IP_SHIN.UPLIP^SHI^ 
.MTH_INTERP}; 
intranges[] = {1,2.1,1,1,0,1}; 

AFfilehandle audiol = AF„NULL_FILEHANDLE, audio2 = AF_NULL_FILEHANDLE; 

int play_flag; 

/* 

* GLX configuration parameter: 

* Double bufTering 

* color index (default so unspecified) 

* nothing else special 
•/ 

static GLXconHg glxConfig [] = { 
{ GLX_NORMAL, GLX^DOUBLE, FALSE }, 
{ GLX^NORMAL, GLX_ZSIZE. 7 }. 
{ GLX^NORMAL, GLX_RGB. TRUE }, 
{ 0, 0, 0 } 

}; 

/* texture properties for the whole face */ 
float texpropsI[] = { 

TX^MINFILTER, TX.POINT, 

TX^MAGFILTER, TX^POINT, 

TX_WRAP. TX_CLAMP, TX_NULL}; 
/* texture properties for the lips ♦/ 
float tcxprops2[] = { 

TX_MINFILTER, TX_BILINEAR. 

TX_MAGFILTER, TX^BILINEAR, 

TX_WRAP, TX_CLAMP. TX_hrULL}; 

Texture environment for the face and the lips */ 
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noat tevpropsU] - {TV_^DECAL, TV_NULL); 

void Refiesh(int dl, int d2) 

{ 

if (RedrawNccded) { 
rccomputefaceO; 
DrawSceneO; 
RedrawNecded = FALSE; 

) 

) 

void cancelCB(Widget w, XtPointer xtp, GlxDiawCallbackStruct *call_data) 

{ 

cxit(0); 

} 

void da_initCB(Widget w, XtPointer xtp, GlxDrawCallbackStnict ♦call_data) 
{ 

2buffei(TRUE); 
winconstratntsO; 
reshapeviewportO; 
subpixcl(TRUE); 

iTiinode(MVIEWING); 

ortho(-xsize/2.0+.5,xsi2e/2.(H.5rysize/2.(HJ,ysizc/2,(H.5r500.0,2(HX).0); 

zfar = getgdesc(GD_ZMAX); 

lsetdepth(0, zfar); 

zclear(); 

cpack(bgc); 

clcai<); 

concavc(FALS£); 
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mmode(MPROJECnON); 
getniatrix(inp); 
iiiiiiodc(MVIEWING); 
frontbuffer(TRUE); 
RedrawNccdcd = TRUE; 

} 

void da_rcsetCB(Widget w, XtPointer xtp, GlxDrawCallbackStnict *call_data) 
{ 

RedrawNeeded = TRUE; 

} 

void UIIiiit(int *argc, char **aigv) 
{ 

Widget cancel, foim, rc,rcl, go, sep, label; 
int i; 

XColor color, uu; 
Colormap cmap; 

Pixel bg_color, top shadow, bottom shadow, fg, sclect color; 
XmString frame_str = XinSthngCreateSmiple('*Fraine: '*), 

label^str = XmStringCreatcSimpleC" DEMO**), 

wf^str - XmStringCrcateSimpleC 1/1 "), 

go^str « XmStringCreatcSimpleC* Go!"), 

cancel_str = XmStringCrcaleSimpleCQuit"); 

tl = XtAppInitialize (&ac. "Dub", NULL, 0, argc, argv, hOJLL, >aJLL, 0); 
dpy = XtDisplay (tl); 

form = XtVaCreateManagedWidgetCform", xmFonnWidgetCiass, tl, 
XmNnoRcsize, TRUE, 
NULL); 

da = XtVaCrcateManagcdWidget ("da". glxMDrawWidgetClass, form, 
GlxNglxConfig, glxConfig, 
XmNwidtfa, xsize, 
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XmNheight, ysize, 

XmNbottomAttachmcnt, XmATTACH^FORM, 

XmNtopOCfset, 2, 

XjnNbottomOffset, 2, 

XmNbackground, WHITE, 

NULL); 

XtAddCallbackCda, GlxNresizcCallback. (XtCallbackProc)da_rcsctCB, NULL); 
XtAddCallback(da, GlxNexposeCallback, (XtCallbackProc)da_resetCB, NULL); 
XtAddCa!lback(da, GlxNginitCallback. (XtCallbackProc)da_mitCB, NULL); 
XtRealizeWidget (tl); 

) 

void Ixiit(tnt argc, char •*argv) 
{ 

char vbufF[50]; 

long zbuf = getgdesc(GD^BITS_NORM_ZBUFFER); 
long rgb_sbits[3], rgb_dbits[3]; 

int i, ch, sourcc_movie_naine_flag = 0, trans_movie_name_flag = 0,last_franie_flag==0; 
int n^frames.pnn; 

char model_filenaine[MAX_NAME]; 

char tcmp[MAX_NAME], paramstMAX_NAME]; 

xsize = getgdesc(GD_XPMAX); 

if (argc < 5) 
ProgUsage(argv[0]); 

/♦ See if we're on a GT. ♦/ 
gversion( vbufT ); 

rgb_sbits[0) - gctgdesc(GD_BITS_NORJM_SNG_RED); 
rgb_sbits[l] = gctgdesc(GD_BITS_NORM_SNG^GREEN); 
rgb_sbits[2] = gctgdesc(GD_BITS_NORM_SNG_BLUE); 
rgb_dbits[0] getgdcsc(GD_BrrS_NORM^DBL_RJED); 
rgb,dbits[l] = gctgdcsc(GD_BITS_NORM_DBL_GREEN); 
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rgb_dbitst2] = gctgdesc(GD_BITS_NORM^DBL„BLUE); 
ok_for_shading= ( 

(rgb_sbits[0]>0) && 

(rgb^sbits[ll>0) && 

(rgb_sbits[2]>0)); 

ok_for_double_bufrering- ( 

(rgb_dbits[0]>=OK_FOR_DOUBLE_BUFFERING) && 
(rgb_dbits[ 1 ]>=OK^FOR_DOUBLE^BUFFERING) && 
(rgb^dbits[2]>=0K^F0R,D0UBLE_BUFFERING)); 

if^debug) { 

fpnntf][stdout/*Machine is %s at res %ld\n*',vbuff,xsize); 
fpnntf(stdout,'*zbuner bits is %ld\n'\zbuf); 

fprintf(stdout;'rgb single is %ld,%ld.%ld\n",rgb_sbits[0].rgb_sbits[l),rgb_sbits(2]); 
fprintf(stdout,"rgb double is %ld,%ld,%ld\n",rgb_dbits[0],rgb_dbitsllj,rgb_dbits[2]); 

} 

/* initialize some global s */ 
prefix = "DATA/face^modcl"; 
texture_map = FALSE; 
param_face = TRUE; 
eye_positionnot_set = TRUE; 
show model = TRUE; 
lips_only = FALSE; 
put_image = TRUE; 
which^image = FRONT^IMG; 
showjeeth = FALSE; 
showcoord = TRUE; 
wire = TRUE; 
one^frame^mode = FALSE; 
mouth^^area = TEX_NONE; 
normalize_mouth = TRUE; 
verbose_modc = FALSE; 
current^window = MAIN; 
curr_savcd_iiiiage « 1; 
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RcdrawNceded = RecalcNceded = TRUE; 
mouth_scale = 1.0; 

first^framc = translator.first^frame = DEF_FIRST_FRAME; 
strcpy(output_movie_name, DEF_OUTPUT_MOVIE_NAME); 
params[0] = 0; 
jpeg_quality = 93; 
play^flag = FALSE; 
bgc = 0x00202020; 
1; 

tex_modc = DUB^MODE; 



whilc(-argc && (ch = argv[!++][l])) { 
switch (ch) { 
case 's': /♦ Source movie */ 
strcpy(source_inovie_naine, argv[i++]); 
source_niovie_iiame_flag++; 
break; 

case 'o': /* New movie */ 

strcpy(output_movie__namc, argv[i-H-]); 

break; 
case 't*: 

strcpy(trans_movie_name. argv[i-Hf)); 
trans_inovie_iiame_flag-H-; 
break; 
case *u': 

strcpy(force_fraines_filenaine, argv[i++]); 

force_frames_flag-H-; 

break; 
case T: 

first^frame = atoi{argv[i++]); 

break; 
case T : 

last_firame = atoi(argv[i-M-]); 

last_fTame_flag++ ; 

break; 
case 'q': 
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jpeg_quality = atoi(argv[i-H-]); 
sprimf{teinp, "-q_%d" Jpeg^quality); 
5trcat(parains,teznp) ; 
break; 
case 'm': 

if (stnicmp(argv[i-l],"-mc",strlen("-mc")) =0) { 
sscanf(argv[i++],"%lx",&bgc); 
printf("Mouth color = %x\n",bgc); 

} 

else ( 

mouth_scale = atofi[argv[i++]); 

sprintfttemp,"-m_%.2r,mouth_scale); 

strcat(params,temp); 

} 

break; 
case T': 

translator_first_frame = atoi(argv(i++]); 

break; 
case 'r': 

frame^rangc = atoi(argv[i-H-]); 

sprintfl[temp,"-r_%d*\fTame_raoge); 

strcat(params,temp); 

break; 
case *w*: 

strcat(parains, "-w"); 

wire = TRUE; 

putjmage = FALSE; 

argc++; 

break; 
case 'a*: 

if {argv[i-l][2] ==== 'p*) avg_j)arains_flag = TRUE; 
else if (argv[i-l][2] = V) avg_width_flag = TRUE; 
strcat(params,argv[i- 1 ]); 
argc-H-; 
break; 
case 'b*: 
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full^flag = TRUE; 
argc++; 
break; 
case *n': 
normalize^mouth = FALSE; 
argc++; 
break; 

case T: /* use INside of mouth ♦/ 
if (argv[i-l][2] = 'N^ mouth_arca = TEX^IN; 
strcat(parains,"«IN**); 
argc++; 
break; 

case 'E': /• use EXTERNAL mouth ♦/ 
if (!stmcmp(argv[i-l]+2, "XT".2)) mouth^arca = TEX_EXT; 
strcat(paTams."-EXT"); 

argc++; 
break; 

case 'O*: /* use OUTside of mouth */ 

if (argv[i-l)[2] == *U*) mouth_area = TEX_OUT; 

strcat(params,"-OUT"); 

argc-H-; 

bresik; 
case *v': 

verbose_mode = TRUE; 

argc++; 

break; 
default: 

fprint£{stdciT, "%c is an illegal flag!\n", (char)ch); 

argc++; 

break; 

) 

argc-s 
} 

if (!source_movie_name_flag || !trans_movie_name) ProgUsage(argv[0]); 
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inake_face(); /* read data, topology and parameters. ♦/ 
if (!put_image) { 



} 

objcctJs_visible[FLESH] » TRUE; 
object_is_visible[LIPS] = TRUE; 
object_is_visible[EYELASH] = TRUE; 
object_is_visible[TEETH] = FALSE; 
object_is_visible[PUPIL) = FALSE; 
object Js_visible(IRIS] = FALSE; 
objectJs_visibIe[FRINGE] = FALSE; 
object Js_visibic(EYEWHrrE] = FALSE; 
setPolarAziinuth(O.O); 
setPolarInc(90.0); 
setPolarTwtst(O.O); 

saved_front_sc = savcd_sidc_sc = sc; 

sprintfl[niodel_filename,"%s.niode!",sourcc_movie_name); 
restareModel(model_filename); 

sprintf(teinp, "%s.%d.prm", source^movie name, first frame); 
PRINTFl ("Restoring params from %s\n", temp); 
restoreParameters(temp,leftj5arameter,rightjparameter,FALSE); 

mth_color = 01; 

/* Create output movies and open input movies */ 
if ((movie = MOVIE_open(source_movie_namc)) = NULL) { 
cxit(0); 

) 

xsize = MOVIE_fTameWidth(movie); 
ysize = MOVlE_framcHeight(movie); 
printfl["Source frame size is (%dx%d)\n'\xsize,ysize); 

if ((trans_movie = MOVIE_opcn(trans_movic_name)) ~ NULL) { 



fgc^OxFFOOOOOO; 
bgc=OxFFFFFFFF; 



/♦ set background color. */ 
/* set foreground color. */ 
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cxit(0); 

} 

sprintfl[audio__rile_namc/'%s.a",trans_inovie_name); 
if{ftilLnag) { 

sprintf{full_^dubbed_movie__name/•%s%s_FULLSIZE^output__movie_name,para^ 
printffCreating movie: %s.mv\n",fiill_dubbcd_movie_namc); 
out^movie 1 = MOVIE_crcatc(fijlI_dubbed_movie_naine,xsi2e,ysize, 

MOVIE_fratneRatc(trans_movic), 

MOVIE_audioRatc(trans_movic). 

DM^TOP^TO^BOTTOM, 

DMJMAGE_JPEG. 

DMJMAGE_^INTERLACED_EVEN, 

jpcg^quality); 

audio 1 = MOVIE_crcatcAudioTrack(audio_file_nainc,out_movicl); • 

) 

else { 

sprintf(dubbed_niovie_narae, "%s%s", output_movi e_name,params); 
printf("Creating movie: VoS.mvVn^.dubbed movie^name); 
out_movie2 = MOVIE_crcate(dubbcd_movie_iiamc,xsize/2,ysizc/2, 

MOVIE_frameRate(trans_movie), 

MOVIE_audioRate(trans_movie), 

DM_BOTTOM_TO_TOP, 

DMJMAGE^MVCl, 

DMJMAGE^NONINTERLACED, 

jpeg^quality); 

audio2 = MOVIE_creatcAudioTrack(audio_file_name,out_movie2); 

} 

if (llast frame flag) last framc = MOVIE_numOfFrames(movie)- 1 ; 
if (MOVIE_numOfFramcs(movie) = 1) onc_framc_mode = TRUE; 

curr^framel - GimgCreate (xsize,ysize,GRGBA); 

n_frames = last_frame + I; 

MALLOC(left_source_array,n_fhimcs,Paramcter_ptr); 
MALLOC(right_source_aiTay,n_frames,Parameterj)tr); 
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restoreAllParameters(source inovie_name,n_&ames, 

left_sourcc_array,right_sourcc_anay); 
MALLOC(lcft_trans_anay,n_fnimcs,Paramctcrjptr); 
MALLOC(right_trans_array,n_fraines,Paramctcrj)tr); 
restoreAllParamcters(traiis_movie_namefn_frames, 

lcft_trans_aiTay,right_trans_aiTay); 

if (vcrbose^mode) { 
for (i=0; i < XtNumber(which_paraiiis); i++) { 
prm = which_params[i]; 
printfl["%s\t",left_parainetcr[prm].boxJabel); 

) 

> 

) 

float calcParamDist (int fraine_no) 
{ 

float distance'^O, min_dist « SOME BIG^NUMBER, t[5], range; 
Parameter ♦p_n, ♦p_l2, *p_rl, *p_r2; 
int k; 

left_source_array[frame_no]; 
rightsourcearrayfframeno]; 
left_paranieter; 
right_parametcr; 

range = MAX_VAL(pJl[JAW_ROTATION])-MIN^VAL(pJlfJAW^ROTATION]); 
t[0] = (VAL(pJl[JAW_R0TATI0N])-VAL(pJ2[JAW_R0TATI0N]))/rangc; 

range = MAX_VAL(pJl[MTO^Yscl])-MIN_VAL(pJl[MTH_Yscl]); 
t[I] = (VAL(pJl[MTH_Yscl])-VAL(p_12[MTH_Yscl]))/range; 
t[2] = (VAL(p^rl(MTH^YscI])-VAL(p_r2lMTH^Yscl]))/range; 

range = MAX^VAL(pJl[RSE^UPLIP])-MIN^VAL(p^ll[RSE_UPLIP]); 
t[3] = (VAL(p_ll[RSE_UPLIP])-VAL{pJ2[RSE_UPLIP]))/rangc; 



pjl = 
p_rl = 
pJ2 = 
p r2 = 
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range = MAX^VAI^^ll[LWRLIP_FTUCK])-MIN_VAL(pJl[LWRLIP^^ 
t[4] = (VAL(pJl[LWRLIP_FTUaC])-VAL(pJ2[LWRLIP.FTUCK]))/^^ 

for (k=0; k<5; lc++) distance fabs(t[kl); 

return (distance); 

} 

int dist_comp (const void *dl, const void *d2) 
{ 

float diff = ((dist_st *)dl)->distancc - ((dist_,st ♦)d2)->distance; 
if(diff< 0.0) return -1; 
else if (diff > 0.0) return 1 ; 
return 0; 

} 

int angleTooFar (int a) 
{ 

Parameter *ls = lcft_source_array[a], ^Ip = leftjiarameter; 

float twist 1 = VAL(!s[POLAR_TWIST]), twist2 = VAL(lp[POLAR_TWIST]); 

if (twist I >= 180.0) twistl 360.0; 
if (twist2 >= 180.0) twist2 -= 360.0; 

if ((fabs(VAL(ls[POLAR_AZIM])-VAL(!p[POLAR^AZIM])) > 10.0) || 
(fabs(VAL(ls[POLAR^INC])-VAL(Ip[POLAR^INC])) > 10.0) || 
(fabs(twistl-twist2) > 10.0)) { 
renim (TRUE); 

} 

return (FALSE); 

} 

int scaleTooFar (int a) 

{ 
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Parameter *ls = lcft_source_aiTay[a]. ♦Ip «= lcft_parametcr; 
float ratio = VAL(ls[SCALE]yVAL(Ip[SCALE]); 

if (ratio < 0.9 || ratio > LI) return (TRUE); 
return (FALSE); 

) 

int loadMouth (int frame^no, int last^chosen) 

int i. j, k, best^fit = SOME_BIG_NUMBER. width,height,in(icx = SOME_BIG_NUMBER, 

closest, found 1, bigger^mouth; 
int from = MAX{fu3t_fTranic,ftamc_no-frame_range), 

to = MIN(last_franic,frame_no+frame_range); 
float distance, min^dist = SOME_BIG__NUMBER; 
char temp[MAX_NAME]; 
Gimg sub_img; 
dist_st distances; 

for (i=from; i < to; i-H-) { 
distance = calcParamDist(i); 
distance /= fsin(RAD( 1 .5*(i-last_chosen)H-90.0»; 
if (distance-inin_dist<-0.00000 1 ) { 

min_dist ~ distance; 

best^fit i; 

printfl['Tramc#%d: best fit is franie#%d {%f)\ii\ frame_no,best_fit,niin_dist); 

MOVIE _getFrame (movie, best_fii,curr_framcl); 

sprintf(temp/'%s.%d.tex",source_movie_naine, bcst_fit); 
restoreTexCoord(temp,left_mthj3ts,right_mthj)ts, 

left_mth_tex,right_mth_tcx); /* Also sets model region vars */ 
recompute_face(); 

width = (int)(right_model-lcft_model); 
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height = (intXtop^model-bottom.model); 

subjmg = GimgCrcateSubCopy(curr_^framel,(int)Ieft^model, 

(int)bottom_modcl,width,hcight); 
tevdef(l, 0, tevpropsi); 

GimgSetAipha(sub_img,255 1) ; 

texdcf2d{2, 4, width,height, (ujong *)ImgRast(subJmg), 7, texpropsl); 
GimgSetAlpha(sub_img,255l); 

tcxdef2d(3. 4. width,height. (uJong •)ImgRast(sub_img), 7. texpropsl); 

GimgDel(sub_img); 

return (bcst_fit); 

} 

int insertMouth (int frame_no, int best_fit) 

{ 

int i, j, k,width,height; 
char temp[MAX^NAME]; 
Gimg sub^img; 

printf("Framc#%d: mouth taken from franie#%d\n'*, frame no, bcst_fit); 

loadActoTLipsParameters(best_fit); 

DrawSceneO; 

MOVIE^gctFrame (movie, best_fit,curr_£ramel); 
sprintf(temp,"%s.%d.tex",source_movie_name, best_fit); 
restorcTexCoord(tcmp,lcft_mthj3ts,iight_mthj>ts, 

left_mthjex,right_mth_tex); /♦ Also sets model region vars ♦/ 
recomputefaceO; 

width = (int)(right_model-left_model); 
height - (int)(top_model-bottom_model); 

tevdefll, 0, tevpropsi); 

sub_img =GimgCreateSubCopy(curT_frame l.(int)left_model. 



SUBSnrUTE sheet (rule 26) 



wo 97/15926 



43 



PCT/1B96/01056 



(int)bottom_niodel,width»height); 
GimgSetA]pha(sub_img,2551); 

texdcf2d(2, 4, width,height, (ujong ♦)ImgRast(sub_ung), 7, texpropsl); 
GimgSetAlpba(sub_iing,2551); 

texde£2d(3, 4, width,heigbt, (uJong *)ImgRast(sub_img). 7, texpropsl); 

GimgDel(subJing); 

return (best_fit); 

) 

^*4t ««« 4! 41 ««««(*«4t4i4<4ii»4i*« 41*41 4rt»«««««*««««««*«^ 

void loadParameters(Parameter *lcft_array. Parameter *right_arTay) 

{ 

int i; 

for (i=0; i<MAX^PARAMETERS; i++) { 
lcft_parametcr[i] - lcft_array[i]; 
right_paramcter[i] = right_array[i]; 

} 

} 

float nonnalizedParam(Paraineter ♦*trans_params. 
Parameter ♦source^params, 
int frame no, 
int prm) 

{ 

float res, V, max_t, nnin_t, min_s, max_s; 

min_t - MIN_VAL(transj)arams[0][prm]); 

max_t = MAX_VAL(trans_params[0][pmi]); 

min_s = MIN_VAL(sourcc_params[pnn])*mouth_scalc; 

max_s = MAX_VAL(source_params[prm])*mouth_scaIe; 

V = { VAL( trans j>arams(frame_no)[prm])-min_t)/(max_t-min_t); 

res = V ♦ (max_s-niin_s) + min_s; 
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return (res); 

) 

void smoothLipsParameter$(int frame^no, int prm, int range, float *1, float •r, 
int normalize_flag) 

{ 

float 11 = 0.0, rl = 0.0. inin_I « SOME_BIG_NUMBER, 

miller = SOME_BIG_NUMBER, inax_l = -SOME_BIG^NUMBER, max_r = 
-SOME_BIG^NUMBER; 
float cur_l, ciir_r; 

int i, from = MAX{first_framc,frame_no-rangc), 
to = MIN(last_frame,fraTne_no+range); 

if (normali2e_flag) { 
cur_l = normali2edParam(left_trans_arrayJeft_source_amy[0],frarae_no,pnn); 
cur_r = normalizedParam(r!ght_trans_array,right_source_array[0],frame_no,pnn); 
for (i=from-framc_no; i <= to-frame no; i-»-+) { 

11 += nonnalizcdparam(left_trans_array»lcftsoiircc_airay[0],framc_no+i,prm); 

rl += iiormalizcdParam(right_trans_array,right_source_arTay[0],fninic_no+i,prin); 

if (II < minj) minj II; 

if (11 >maxj) maxj = 11; 

if (rl < inin_r) min^r = rl; 

if (rl > max_r) max_r = rl; 

} 

} 

else { 

if (prm == MTH^Yscl) ( 
curj = VAL(left_trans_array[fTame_no][pnn])*raouth_scale; 
cur_r = VAL(right_tTans_array[frame_no][prni])*inouth_scale; 

) 

else { 

curJ = VAL(lcft_trans_array[frame_noJ[pnn]); 
cur_r = VAL(right_trans_array[frame_no][pnn]); 

} 

for (i=ironi-framc_no; i <« to-franie_no; i++) { 
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if(prm =«MTH^Yscl) { 
11 VAL(left^trans_aiTay(frame_no+i][pmi])*mouth_^scalc; 
rl += VAL(right_trans_array[frame_no+i][pnn])*mouth_scalc; 

} 

else { 

n VAL(lcft_trans_airay[fraine__no+i][prm]); 
rl += VAL(right_trans_array[firame_no+i][pnn]); 

} 

if (11 < minj) minj = 11; 
if (11 > max_l) max J = 11; 
if (rl < min_r) inin_r = rl; 
if (rl > max_r) max_r = rl; 

) 

} 

♦1 =^ 11 / (to-from-f 1); 
♦t = rl / (to-from+l); 

) 

^♦♦♦♦♦♦•••♦♦♦••♦•♦♦••••♦••••♦•♦♦♦♦♦♦♦♦♦♦♦♦♦♦•♦♦♦•♦♦♦♦♦***********************/ 

void loadTranslatorLipsParameters(int frame_no, int nonnalize j}arains) 
{ 

float Il,12,13,rl,r2.r3; 
int i, pnn; 

int from = MAX(first_fTamc.frame_no-frame_range), 
to = MIN(last_firame,frame_no+frame_range); 

for (i=0; i < XtNumbcr(which_params); i++) { 
prm = which_params[i]; 

if (avgj>arams_flag) 

smoothLipsParamclcrs(frame_no,prm, ranges[i],&12, &r2, normalizejiarams); 
else 

smoothLipsParameters(frame_no,prai, 0, &12, &r2, normalize_params); 
if (prm = MTH_Yscl 1| pmi =-= MTH_INTERP) { 
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if (avg_width_flag) 
VAL(lcftj)arameter(pnnJ) = VAL(right_parametcr(pnn]) = (12+r2)/2.0; 

J 

else { 

VAL(left_parameter[prm]) = 12; 
VAL(right__parametcr[pnn)) = r2; 

} 

PRINTF2("changed to:(%f,%f)\n^VA^leftJ>a^aInetcT[p^n)),VAL(right j)arameter[pnn])); 

if (verbosc_mode) printf{"(%.4f,%.40\t",l2,r2); 

) . 

if (verbose_mode) printf("\n"); 
recomputc_face(); 

} 



void loadActorLipsParameters(int framc^no) 
{ 

int i, pxm; 

float n,12,13,rl,r2,r3; 

for (i=0; i < XtNumbcr(whichj)arams); i++) { 
pim = which_j)arams[i); 

VAL{Ieft_j)arameter[prni]) = VAL(lcft_source_array[fraine^no][pnn]); 
VAL(right_paraincter[prni]) = VAL(right_soiircc_airay[fraine_no][pnn]);; 

) 

recompute_face() ; 



void Go(void) 
{ 

long nof; /• Num of frames ♦/ 

char tcmp(MAX_NAME]. linc[80]; 

int i, j, last, orig_frainc_no,new_framc_no; 
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Gimg small Jmg, grab^frame; 
time_t tm; 

float pl6[2],p288[2], distance,distancel,sb; 
Matrix M; 

FILE •force^framcs; 

int coinpressed_image_size, *coinpressed_iinage; 

if (force_fTames_flag) { 
if ((force_franies = fopen(force_fraines_fiIename, V)) == NULL) 

forceframes^flag = FALSE; 
else { 

fgcts (line, 80, forcc_frames); 

sscanf(hne, "%d\t%d", &orig_frame_no, &new_£rame_no); 

} 

} 

. tm = time(O); 

printfl**Start dubbing %d frames at %s",last_fTame-fiTst_frame, ctime(&tm)); 

rcadsource(SRC_FRONT); 

grab_lTame - GimgCreate(xsize,ysize,GRGBA); 

MOVlE setCurrentFrameNo (movie, first^frame); 
if (one_frame_mode) { 

texturcMapping(TEX_ON); 

DrawSceneO; 

} 

cpack(bgc); 

for (i=first_firame. j=translator_first_frame; i <= last_frame; i++ { 
tcx_mode = TRACK_MODE; 
sprintf1[temp, "%s.%d.prm", source_movie_namc, i); 

if (lone frame mode) loadParameters(Icft_source__arTay[i],right_source_arTay[i]); 
/* else loadParametcrs(left_squrcc_axTay[0),right_source_array[0]);*/ 
else { 

VAL(leftj)arameter[BEARD_SCL]) = VAL(righlj>aramctcr[BEARD_SCL]) = 0; 
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recoinpute_face(); 

) 

VAL(lefit_parameter[BEARD_SCL]) = VAL(right_paranicter[BEARD_SCL]) = 0; 

recomputefaceO; 

DrawSceneO; 

prepareMatnx(M); 

if (putjmage && !one_iraine_mode) { 
tcxturcMappmg(TEX_ON); 
DrawSceneO; 

} 

loadTranslatorLipsParameters(i,normalize_mouth): 
DrawSceneO; 

caIcProjection(left_trans_pts.right_trans_pts); 
if (inouth_area >= TEX_B^) { 
if (forcc_f3ramcs_flag <&& (i = orig_franie_no)) { 

last = inscrtMouth(orig_fiame_no,new_frame_no); 

fgets (line, 80, force^frames); 

sscanfKline, "%d\t%d", &orig_frame_no, &ncw_fiamc_no); 

) 

else { 
if (i == nrst_fTame) 

last = loadMouth(i,i); 
else 

last = loadMouth(i,last); 

) 

) 

else 

printfrTramc#%d\n". i); 

tex_mode = DUB_MODE; 
recompute_face(); 
prepareMatrix(M); 
DrawSceneO; 
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lrectiead(0,O,xsizc- 1 ,ysize- 1 ,(u_long *)ImgRa$t(gTab_frame)); 
MOVIE_addCuiTcntFraine(out_movic 1 ,grab_frame,0); 
MOVIE_addAudioFniine(audiol,out_inoviel); 

) 

else { 

smallimg = GimgResi2e(grab_frame,xsize/2,ysizc/2,2); 
MOVIE_addCuiTentFraine(out_movie2,small_img,0); 
MOVIE_addAudioFranie(audio2,out_inovie2); 
GimgFrce(sinall;^img) ; 

) 

if (put_image && !oiie_fi:anie_niode) { 
MOVIE_nextFraiiie(niovie); 
texnireMapping(TEX_OFF); 

} 



if(fulLflag) { 
MOVIE_close(out_movie I ); 

out^moviel = MOVIE_opcn(full_dubbcd_inovie_name); 

} 

else { 

MOVIE_closc(out_movic2); 

out_movie2 = MOVIE_open(dubbcd_inovic_name); 

) 

tm = tirae(O); 
AFclosefi]e(audio 1 ); 
AFcloseftle(audio2); 
prmtf("Fmished at %s", ctime(&tm)); 

} 

/«««,«««»«*,»,,«**,«^,«,«**«***«**««,, 

void DrawScene(void) 
{ 

zclear<); 

update_window_display(); 

} 
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void ProgUsage(char ^filename) 

{ 

fpnntf)[stderr/'Dub: fix lips in a dubbed movie.\n"}; 

lprintfl;stderr,"Usage:\n"); 

iprintfl[stdeiT," %s [options]\n", filename); 

fprintfl[stderr,"\n Options:\n"); 

fphntf](stderr," -s <movie file>\tSource: movie to be dubbed\n"); 

fpnntf1[stderr," -o <movie file>\tOutput movie name (default«\"movie\")\n"); 

fprintf(stdcrr," -t <movie file>\tTranslator's movie\n"); 

fpnntf(stderT»" -u <filename>\t\tForce usage of info in filename to plant mouth\n"); 

Iphntftstderr," -f <integer>\t\tFir5t frame num, starting at 0 (default)\n*'); 

fprintftstderr," -1 <integer>\t\tLast frame num (dcfau!t=0)\n"); 

fprintfl[stdciT," -F <intcgcr>\t\tTranslator's First frame num (default=0)\n"); 

fprintf(stderr," -q <integer>\t\tJpeg quality (default=93)\n"); 

fprintflstderr," -m <integer>\t\tMouth scale factor (optional)\n"); 

fprintf(stderr," -r <integcr>\t\tRange of frame to search mouth (defalut=25)\n"); 

fprintf(stdcrr," -w \t\t\tCrcate only Wire frame animation (optional )\n*'); 

fi3rintf][stderr," -ap \t\t\t Average params (optional)\n"); 

fpnntf(stderr," -aw \t\t\tAverage mouth width (optional)\n"); 

fpnntf(stden-»" -b \t\t\tCreate a frill size movie (optional)\n*'); 

fjprintftstdcrr," -n \t\t\tDon't Nonnalize mouthVn"); 

fprintf(stderr." -IN \t\t\tUse inside of mouth (dcfault)\n"); 

fprintflstderr," -OUT \t\t\tUse outside of mouthNn"); 

fiprintflstderr," -EXT \t\t\tUse bigger area around mouth\n"); 

fiprintftstderr/* -mc \t\t\tlnside mouth color\n"); 

fpriniflstden/* -v \t\t\tVerbose mode\n*'); 



exit(O); 

} 



main(int argc, char **argv) 
{ 

XEvent e; 
char temp[200]; 
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Init(aigc, argv); 

UIIiut(&argc, argv); 

drawjjoly = 1; 

MOVIE_bind(movie, da); 

MOVIE_setCuiTentFranieNo(movie,first_frame); 

drawfaceO; 

RedrawNeeded = TRUE; 
while (XtAppPending(ac)){ 

XtAppNextEvcnt (ac, &c); 

XtDispatchEvent (&c); 

} 

Go(); 

printfl["Done!\n"); 

I 

♦ Module: general. c 

♦ By: Ranco 

♦ History: 

♦ 18.8.94: creation. 

#include "x_inc,h" 
^include "mv.h" 
#include "gencraLh" 

Set the proper GL matrices: The transfonnation that a point p experience 
is as follows: 

p * (S * Rx(-90)*Ry(-90)*Rx(90-x)*Ry(-y)Rz) ♦ TRANS * Tnew = (U\ V) 

void setMatnces(void) 
{ 

loadniatrix(idxnat); 

translate (left_parameter(HEAD_Yofr|.value, 
left__parameter[HEAD_Zof!] .value, 
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left_parainctcr[HEAD_Xoff].valuc); 
rotateAndScaleO; 

} 



void rotate AndScate( void) 
{ 

/* Global transformation: ♦/ 
/* multmatnx(global_trans);*/ 

rotate((int)(leftj3aramcter[POLAR_TWIST].value*10),'z'); 

rotatc(-(int)(lcft j)aramctcr[POLAR_AZIM].vaIuc* 10)/y'); 

rotate(900-(int)(Icftj)arainctcr[POLAR_INC].value*10)/x'); 

rotatc(-900/y'); 

rotate(-900/x*); 

if (which^image = SIDEJMG) 

scale{savcd_side_sc,savcd_sidc_sc,savcd_side__sc); 
else 

scale(sc,sc,sc); 

) 

void get_compo5ite_matnx(Mathx M) 
{ 

Matrix mv; 
register int ij,k; 

mmode(MVIEWING); 
gctmatrix(mv); 

/* form product mv * mp .... viewing followed by projection */ 
for (i=0; i<4; i++) { 
for (k=0; k<4; k++) { 
M(i][k] = 0.0; 

for (j-0; j<4; M[i][k] += mv[i][jl • mp[j][k]; 

} 
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} 





/* Tipically, both functions above should be called together, and this is */ 
/* what this function does... •/ 

void prepareMathx(Matrix M) 
{ 

sctMatricesO; 

get_coinposite_matrix(M); /* View x Projection matrix •/ 



} 



^^^^^♦^'^^♦^♦•^i^ World to screen coordinates ••**••*♦***♦••••**•♦•♦***•♦•*•*♦/ 

void world2screen(Matnx M, float x, float y, float z, float *sx, float *5y) 
{ 

float v[4); 
register int i; 

long xs = xsize, ys = ysize; 

for (i=0; i<4; v[i] = x*M{0]ti]+y*M[l][i]+z*M[2][i]+M[3][i]; 

if(v[33=ao) { 

Bug("w2s: v[3] = 0.0"); 

fyrintflCstderr, "V (%f, %f, %f. %f)\n", v[0], v[l]. v(21. vt3]); 

return; 

) 

♦sx = ((float)xs/2.0)*(v[0]+1.0)..5; 
•sy = ((float)ys/2.0)*(v[l]+1.0)-.5; 

) 

/************ Returns the distance between 2 points ♦•♦♦•♦♦•****••♦♦♦*♦••♦*•/ 
float dist (float *pntl, float ♦pnt2) 

{ 

return (fsqrt((patl[0].pnt2[0]) * (pntl[0]-pnt2[0]) + 



SUBSTITUTE SHEET (RULE 26) 



wo 97/15926 PCT/IB96/010S6 

54 

{pntl[l].pnt2[I]) • (pntl[l)-pnt2(l]))); 

} 

float hdist (float *pntl, float *pnt2) 
{ 

return (fabs(pnt1[0>pnt2[0])); 

) 

float vdist (float *pntl, float *pnt2) 
{ 

return (fabs(pntl[l]-pnt2[I])); 

) 

/* Matrix utilities */ 

void copy_matrix (Matrix to. Matrix from) 
{ 

register int i, j; 

for (i=0; i<4; i++) 
for ti=0; j<4; j++) 
totiim = fromtilQ]; 

} 

void pnnt_matrix(char "^str. Matrix m) 

{ 

register int i, j; 

printf("Print matrix: %s\n", str); 
for (i=0; i<4; i++) { 
for 0=0; j<4; { 
printf("%3.5f ",m[i][j]); 

} 

priIltf(*^n"); 

} 
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void setPolarTwtst(f1oat val) 
{ 

left_parameter[POLAR^TWIST].value = 
right_parameter(POLAR_TWIST].value = val; 

) . 

void setPolarAziinuth(float val) 
{ 

leftj)aranictcr[POLAR_AZIM].valuc = 
right j>aramcter[POLAR_AZIM]. value = val; 

} 

void setPolarlnc(float val) 

{ 

left_parameter[POLAR_INC).vaIue = 
right_paranieter[POLAR_INC).vaIue = val; 



void addDist2Face(float delta) 



sc delta; 



void addXof!(float delta) 

{ 

left_parameter[HEAD_Xoff|.value = 
right_j)arameter[HEAD__Xoff]. value delta; 

left_paramcter[HEAD_Xofr).changed = 
right jjaramcterfHEAD^Xofflxhanged 1; 

} 
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void addYoff(float delta) 
{ 

leftj)arameter(HEAD_Yofr|.valuc = 

right_parameter[HEAD_YofT|.value +^ delta; 
left_paramcterIHEAD_Yofl]. changed = 

right_paranieter[HEAD_YofT|.changcd - I; 

} 

void addZofflfloat delta) 

{ 

left_parameter[HEAD_Zoff|.value ^ 

right_paraineter[HEAD_Zoff).value delta; 
lcft_parameter[HEAD_Zoff|.changed = 

right_parameter[HEAD_Zoffl. changed = 1; 

} 

void addPolarTwist(float delta) 
{ 

lcft_paramctcr[POLAR_TWIST].value = right_parameter[POLAR_TWIST]. value += delta; 
lcft_paraineter[POLAR_TWIST].changcd = right_parameter[POLAR_TWIST]xhanged =1; 

} 

void addPolarAzimuth( float delta) 
{ 

left_pwmeter[POLAR_AZIM] .value = right_parameter[POLAR_AZIM].value += delta; 
left j)arametcr[POLAR_AZIM], changed = right j)arameter[POLAR_AZIM]. changed =1; 

) 

/ 

void addPolaTlnc(fIoat delta) 

{ 

Ieft_parameter[POLAR_INC].value = right_parameter[POLAR_INC].value += delta; 
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Icft_parameter[POLAR_INCl.changed = right j)aramctcr[POLAR_INC].changcd =1; 

} 

void saveTexCoord(char *filcname, int np) 

{ 

FILE T; 

char temp[MAX_NAME], param_file_namc(MAX_NAME]; 
int i; 

Matrix M; 

float p[2],width,height; 

calcTextureCoords(left_tex,right_tex); 

sprintf][tenip,'*%s.tex'*,filenanie); 

unlink(temp); 

if ((f = fopen(temp, V')) = NULL) { 
fprintftstderr, "Can't open texture file '%s* for writing!", temp); 
return; 

} 

fprintfl[f, "Num of points: %d\n",np); 

fprintfl[f,"Region: %f, %f, %f, %An", left^model, top^model, 

right model, bottom_model); 
width = (floor){right__model-left_model); 
height = (floor)(top_mode1-bottom_model); 
prepareMatrix(M); 
for(i=l; i<=np; i++) { 

world2scrcen(M, leftj)t[i][0]. lcftj>t[i][l), left_ptli][2]. &p[0], &p[l]); 

p[0] = (p[0]-lcft^model)/width; 

p[l] = (p[l]-bottom_model)/height; 

fTprintf(f, "%d\t%f, %f\n'\i,p[0], p[l]); 

world2screen(M, right j3t[i][0], rightj)t[i](l], rightj)t[i]t2], &p[0], &p[l]); 
p[0] = (p[0]-left_model)/width; 
p[ n = {p[ 1 ]-bottom_model)/height; 
fphnif{f, "%d\t%f. %fai",i,p[0],p[l]); 

} 

$etPolarlnc(90.0); 
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sctPolarTwist(O.O); 
setPolarAziinuth(0.0); 

sc = 0.35; /♦ Usually it is around this number... ♦/ 
prepareMatrix(M); 

fprintf(f, "# Normalized projection of selected points of niodel:\n*'); 
for(i=49; i<=70; i++) { 

world2screen{M, leftj3t[i][0]. left^t[i][l], leftj)t[i]l2], &p[0], &p[l)); 

fprintf][f, "%d\t%d, %d\n",i.(int)p[0]. (int)p[l]); 

world2screen{M, rightj>t[i][0], rightj)t[i][l], right_pt[i][2], &p(0], &pll]); 
fprintf(f, "%d\t%d, %d\n",i,(int)pfO], (int)pfl]); 

) 

i = 287; 

world2screcn(M, lcft_pt[i][0], lcftj)t[i][l]. Ieftj>t[i][2], &p[0]. &p[l]); 
ftjrintflf. "%d\t%.Of, %0f\n",i,p(0]+.5, p[l]+.5); 

world2screcn(M, right j>t[i][0], right j>t[i][l), right j)t[i][2], &p[0], &p[l]); 
fprintf(f. "%d\t%Of, %Of\n",i,p[01+.5, p[l}+.5); 

fclose(0; 

sprintf(parani_file_name, "%s.%d,prm", 
out_base_name,(int)MOVIE_getCurrentFrameNo(movie)); 
restorcParameters(param_rile_nameJeft _parametcr,right_parametcr,TRUE); 

) 

void restoreTexCoord(char ^filename, 

float lcft_pt[][2), float right_pt[][2], 
float lcft,tcx[][2], float right_tcx[][2]) 

{ 

FILE ♦f; 

char temp[MAX_NAME].line[MAX_LINE); 

int i, j, np; 

float width,height; 

if ((f = fopcn(filenamc, V)) = NULL) { 
sprintf(tcinp,"%s.tex",filenamc); 
if ((f = fopen(temp, "r")) = NULL) { 
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fprintf(stderr, "^Can't open texture file *%s' for reading!**, filename); 
return; 

} 

I 

fgcts(line.MAX^LINE,0; 

sscanfl[Hne, "Num of points: %d\n*',&np); 

fgets(line,MAX_LINE,f); 

sscanfllinc. "Region: %f, %f, %f, %f\n", 

&left_model, &top_model, &right_model, &bottom_model); 



width = (floor)(right_model-left__model); 
height = (floor)(top_model-bottom_niodel); 
for(i=l; i<=np; 1++) { 
fgcts(line,MAX_LrNE,0; 

sscanftline, "%d\t%f. %f\&j,&left_tex[i][0].&left_tcx[i][l]); 
fgets(hne,MAX_LINE,0; 

sscanfl[line, "%d\t%f, %f\&j,&rightJex[i][0].&iight_tex[i][l]); 

\ 

fgcts(line,MAX^LrNE,0; /* Remark line ♦/ 
while (!feofl[0) { 

fgets(line,MAX^LINE,f); 

sscanflline, "%d".&j); 

sscanfOine. "%*d\t%f, %f\&leftj)t[j][01,&left_pt[j][l]); 
fgcts(line,MAX_LINE,0; 

sscanflline, "%*d\t%f, %r,&rightj)tO][0],&rightj)t[i]Il]); 

) 



fclose(f); 

} 

/• Filename should be added ".mthl" and ".mth2" ♦/ 
/♦ Then, inside of mouth (without lips) coordinates arc saved in .rathl */ 
/* file, and outside of the moith is saved in .mth2 file */ 
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void saveInMth(char ^filename) 

{ 

FILE ♦f; 

char temp[MAX_NAME]; 

/• Save inside of mouth */ 
spriiitf(temp, "%s.mthr', filename); 
unlink(temp); 

if ((f = fopcn(tcmp. V)) = NULL) { 
fisrintf^stdeiT, "Can't open mth param file *%s* for writing!", temp); 
return; 

} 

fprintf(f, "%f, %An".rightjex[287][0], rightjcx[287][l]); /* 0 */ 
fiprintf(f, "%f, %f\n",rightjex[63][0]. rightjex[63][l]); /• 1 ♦/ 
fprintfi[f, "%f, %f\n".rightjex[62][0], rightjex[62][l]); /♦ 2 ♦/ 
fprintf(f, "%f, %f\n".rightjex[61][0], rightjex[61][l3); /♦ 3 */ 
fprintfl[f, "%f, %AnMeftjcx[62][0], leftjex[62][l]); /* 4 */ 
fprintf{f, "%f, %fjiMeftjex[63][0), lcft_tcx[63][l]); /♦ 5 ♦/ 
fprintflf, "%f, %nn",Ieftjex[287](0], leftjcx[287][l]); /♦ 6 */ 
fprintf^f, "%f, %An",leftjcx[60](0], left_tex[60][l]); /* 7 ♦/ 
fprintf(f, "%f. %f\nMeft_tex[59][0]. leftjex[59][l]); /* 8 ♦/ 
fprintf(f; "yof, %AnMeft jcx[58][0], left Jcx(58][l )); /• 9 ♦/ 
fiprintfltf, "%f. %f\n",rightjcx[59][0], rightjcx[59]tll); /• 10 ♦/ 
fprintf(f, "%f. %An",rightjex[60]I0]. rightjex[60][l]); /Ml ♦/ 
fclosc(f); 

/* Save outside of mouth */ 
sprintf(temp, "%s.mth2", filename); 
unlink(temp); 

if ((f = fopen(temp, "w")) == >njLL) { 
fpnntfKstderr, "Can't open mth param file '%s* for writing!", temp); 
return; 

} 

lprintf(f, "%r, %f\n".right_tex[70][0]. right_tcx[70][I]); /• 0 */ 
fprintf][f, "Vof, %f\n'\rightjcx[69][0], rightjex[69][l]); /• 1 */ 
fprintf^f. "%f, %f«",rightjcx[68]I0], rightjcx[68][l]); /• 2 •/ 
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fprintfl[f. "%f. %fm".rightjex[67][0]. rightjex[671tl]); /* 3 •/ 

fprintf(f, "%f, %f\nMcftjex[68][0], leftjex[68][l]); /* 4 */ 

fprintf(f. "%f. %f\nMeftjex[69][0], leftjcx[69][l]); /* 5 */ 

fprintflf, "%f. %f\nMeftjext70][0], left_tex[70][l]); /* 6 ♦/ 

fprint£(f, "%f, %f\nMeftjex[51][0], IefUex[51][l]); /• 7 ♦/ 

fprintfl[f, "%f, %f\nMeftjex[50][0], leftjex[50][l]); /♦ 8 ♦/ 

fprintftf, "%f. %f\n",Ieft_tex[49][0], leftjex[49][l]); /♦ 9 */ 

iprintftf, "%f, %f\n",rightjex[50][0], righijex[50][l]); /♦ 10 */ 

fprintfl[f. "Vof, %f\n".righuex[51][0], rightjex[5 !][!]); /* 11 */ 
fclose(f); 

/♦ Save external mouth */ 
sprintfl[tcnip» *'%s.mth3'*, filename); 
unlink(temf)); 

if ((f = fopen(tcmp, "w")) == NULL) { 
fprintf(stderr, "Can*t open mth param file '%s' for writing!", temp); 
return; 

} 

fprintflf. "%f, %fai",rightjex[44][0], righLtex[44][l]); /♦ 0 ♦/ 

fprintfl[f, •*%f, %An",rightjex[73][0), rightjex[73](l]); /* 1 */ 

fprintf(f, "%f. %f\n",right_tex[72][0], rightjex[72][l]); /* 2 */ 

fprintf(f, **%f, %f\n".rightjex[71][0]. rightjex[71][l]); /* 3 •/ 

fprintf(f, "%f, %f\n'Mcftjex[72][0]. leftjcx[72][l]); /♦ 4 */ 

fprintf(f. "yof. %f\nMcftJex[73][0], Icftjex[731[l]); /♦ 5 •/ 

fprintf{f/"%f. %f\nMeftJext44][0], lcft_tex[44][l]); /* 6 •/ 

fprintf(f, "%f, %faiMeftjex{43][0], leftjex(43][l]); /♦ 7 ♦/ 

fprintKf, "%f, %f\n",leftjex[42][0], lcft_tex[42][l]); /• 8 */ 

fprintftC "%f, %AnMeftjex[4I][0], Icftjex[41][l]); 1*9*1 

fprintf(f, "%f. %An",rightjex[42][0), right^tex[42][l]); /* 10 ♦/ 

fprintfl;f, "%f. %An",right_tex[43][0], rightjex[43][l]); /♦ 11 ♦/ 
fclose(f); 



int int_comp (const void *il, const void *i2) 
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{ 

return (♦((int ♦)il) - ♦((int ♦)i2)); 

} 

int readKeyFramelndices (char *filename, int *key_fratnes, int ♦init^framcs) 
{ 

FILE ♦f; 
int i = 1; 

char temp[MAX_LINE]; 

key_frames[0] = init_franies[0] = 0; 
sprintfl[temp."%s.kf',filename); 
if ((f = fopcn(tcmp, "r")) == NULL) { 
return (FALSE); 

} 

while (fgets(tcmp.MAX^LINE,f) && !fcof(f)) { 
sscanf(temp. "%d^ &kcy_frames(i]); 
sscanf(tcnip, "%d^ &init_fTames[i++]); 

> 

fclose(f); 
key_frames[0] = 

qson (key_frames+l,kcy_frames[0],sizeof(int),int_comp); 

sprintf(tcmp;'%s.ir,filcnainc); 
if ((f = fopenCtemp. V)) = NULL) { 
return (FALSE); 

) 

while (fgets(tenip,MAX^LINE,0 && !feof(0) { 
sscanf(tcnip, "yod", &init_frames[i++]); 

) 

fclose(f); 

init_fTaiiies[0] = i-1; 
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qsort (init_fraines+l,init_&aines[0],sizeof(int)»int_comp); 
printf("%d init frames: ",init_fraincstO]); 
for (i=l; i <= init_frames[0]; i++) 

printf ("%d init_,frames[i]); 
printf^'W/od key frames: ",kcy_framcs[0]); 
for (i=l; i <= key_frames[0]; i-H-) 

printf ("%d key_frames[i]); 
printf{"\n'*); 
retuni (TRUE); 

} 

void restoreInMth(char *filename, float into[12][2]) 

{ 

FILE ♦f; 

int i; 

char temp[MAX_LINE]; 

if ({f = fopen{filename. V)) = NULL) { 
return; 

} 

for (i=0; i < 12; i++) { 

fgets(temp.N4AX_LINE.O; 

sscanfftcmp, "%f, %r, &into[i][0], &into[i][l]); 

1 

fclose(0; 

} 

void swapParameters(void) 

{ 

int j; 

if (which_image = STOE JMG) { 
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for 0-0; j < NUM^SAVED; { 
saved_front_params(0][j] = VAL{left_parameter[save_those[j]]); 
saved_front_params[l][j] = VAL(rightjjarameter(save_those[jJ]); 
VAL(leftj5arameter[save_thosc[j]]) = savcd_sidej)arains[0](j]; 
VAL(right_j)arametcr[savc_thosc|j]]) = savcd_side_paranis[l][j]; 

} 

} 

else { 

for (i^O; j < NUM_SAVED; j++) { 
saved_side_params[0]|j] = VAL(leftj)arameter[save_those|j]]); 
saved_side_j)aranis[l}[j] = VAL(nght_parameter[save_those(j]]); 
VAL(leftj3arametcr[save_those|j]]) = savcd_fTontj)arains[0][j]; 
VAL(right_paramcter[savc_thoseO])) - savcd_fTont_params[l](j]; 

} 

} 

I 

/• Converts Gimg (unsigned long (RGB)) image to char ♦result, which is */ 
/* The weighted average of the input image. Expects result memory to ♦/ 
/♦ be allocated. •/ 

void longImg2charVec (u_long 'img, int sx, int sy, int width,int height, int xsize, char ♦result) 

/ 
I 

register int y, x, dx = xsize - width; 
img += sy^xsizc + sx; 



for (y=0; y < height; y-H-) { 
for (x=0; X < width; x++) { 
♦result = (char)greyColor(^img); 
result++; 
img++; 

} 

img += dx; 
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} 

void findRegion(int from_vertex, 
int to_vcrtex, 
float ♦left, 
float *top, 
float *right, 
float ^bottom, 
float w_scalc_factor, 
float h_scale_factor) 

{ 

float ♦u, ♦<!, *I, *!; 
Matrix M; 
float temp; 

float max x, max y, min x, min_y; 

int size, corj, niax_x_i, max_y_i,min_x_i,inin_jrj; 

float width, height,Ip[2],tp[2]; 

max_x « -95>999.0, max^y - -99999.9; 
min_x = 99999.0, iiiin_y = 99999.9; 
prepareMatrix(M); 

for (j=Hroin_vertex; j <= to_vertex; { 
world2screen(M, left_pt[j][0], Ieft_pt[j][l), len_j>t[j][2), &lp[0], &Ip[l]); 
worId2screen(M. right j5t[j][0], right_pt[i][ 1 ]. right_pt[j][2], &ip[0], &rp[l]); 
if (GT(lp[0], max_x)) { max^x = lp[0]; 1 = leftj)t[j];max_xj=j;} 
if (GT(rp[0], niax_x)) { max_x = rp[0]; 1 = righi_pt(j];max__xj==j;} 
if (LT(lp[0], min_x)) { min x = lp[0]; r = leftj)t[j];min__xj=j;} 

' if (LT(ip[0]. min_x)) { min^x = rp[0]; r = rightj5t|j];min_xj=j;} 

if (GT(lp[l]. max^y)) { max_j^ = ip[i]; u = left_j)t[j];max_y^i=j;} 
if (GT(rp[l], max_y)) { max_y = rp[l]; u = right_pt[j];max_yj=j;} 
if (LT(lp[l], min_y)) { min^y = lp(I]; d = icftj)t|j);min_yj=j;} 
if {LT(rp[l]. min_^)) { min^ = ip[l]; d = rightj)t[j];min_yj==j;} 

} 
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world2screen(M, r[0], r[l], r[2], left, Aiemp); 
world2screen(M, u[0], u[l], u[2], &temp, top); 
world2screen(M. d[0], d[l], d[2], &tenip, bottom); 
world2scrccn{M, l[0], 1[2], right, &temp); 

width = (*right) - ('left) + 1; 
height = (*top) - (*bottom) + 1; 
♦left -= width * w_scalc_factor; 
♦right += width ♦ w_scaIe_factor; 
•top += height ♦ h_scale_factor; 
♦bottom -= height ♦ h_scale_factor; 

♦left =MAX(l,(*left)); 

♦right = MIN(MOVIE__firameWidth( movie)-!. (♦right)); 
♦top = MIN(MOVIE_frameHeight{movie)-U(^top)); 
♦bottom = MAX(1, (♦bottom)); 

) 

void setModelRegion(void) 
{ 

findRegion(l,np, 

&left_model, &top_model, 
&right_modcl, &bottom_mode], 
0,0,0.0); 

left^model = floor(left_model); 
right_model = floor(righl_model+.5); 
bottom^model = floor(bottom_model); 
top_model = floor(top_modcl+.5); 

} 

Parameter tcmp_left[MAX_PARA METERS], temp_right[MAX_PARAMETERS]; 
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int restoreLipsParameters(mt frame^no) 

{ 

int i, prm; 

noat n,I2J3,rl,r2,r3; 
char temp[MAX_NAME]; 

int which[] = PAW^ROTATION,Nmi_YscI,RSE.UPLIP,LWRXIP_FTUCK,Mra_I>rre 

sprintf(temp,"%s.%d.pnn",out_base_namc, fnune^no); 
if (!restoreParameters(teinp,tempJeft, tcnip_right,TRUE)) return (FALSE); 
for (i==0; i < XtNumbcr(which); i-H-) { 
prm = which[i]; 

VAL(lcftj5aranictcr[pnn]) = VAL(tcinp_lcft[pnn]); 
VAL(right_parameter[pnn]) = VAL(tcmp_right[pnn]); 

recompute_face( ); 
return (TRUE); 



void restoreModel(char •filename) 
{ 

char tcmp[MAX_LINE]; 
FILE ♦f; 
int i; 
float *p; 

if ({f = fopen(filename. V)) = NULL) { 
prim ft "%s not found!\n'\fi1ename); 
return; 

) 

fgets (temp. MAX_LINE, f); 
for (i=l; i <= np; { 

fgets (temp, MAX^LINE. f); 

p = left_inpts[i]; 

sscanflCtcmp, "%*d\t%At%At%f\ &p[0], &p[l], &p[2]); 
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fgets (temp, MAX^LINE, 0; 
p = rightjnpts[i]; 

sscan£(tcmp, "%*d\t%f\t%f\t%r, &p[0], &p(l], &p[2]); 

} 

fclose(0; 

} 

void restoreAUParameterstcbar ^filename, 
int n_frames, 
Parameter **left_array. 
Parameter **right_array) 

{ 

int i, j,pmi; 

char tcmp[MAX_NAME]; 

for (i=0; i<n_fTamcs; i++) { 
MALLOC(lcfi^array[i],MAX_PARAMETERS,Parametcr); 
MALLOC(right_aiTay[i],MAX_PARAMETERS,Paramctcr); 
bze^o(left_aITay[i],^4AX_PARAMETERS»sizeofl[Pa^ametc^)): 
bzero(right_anray[i],MAX_PARAMETERS*sizcofl[Paramctcr)); 
sprintf(tcmp, '•%s.%d.pm[i*', filename, i); 
restoreParameters(tcmp,lcft_aiTay[i].right_arrayti],FALSE); 

if (i==0 && (n^^frames > 1)) { 
for 0=0; j < NUM^ACTIVE^MTH_PARAMS; { 
prm = activc_mthj3arams[j]; 

MAX_VAL(Icft^aiTay[0][pmi]) = VAL(!eft_arTay[0][pnn)); 
MIN_^VAL(Ieft_array[0][prm]) = VAL(lcft_array[0][pmi]); 
MAX_VAL(right_array(0][pmj]) = VAL(right^array[0](pnn]); 
MIN_VAL(right__array[0](pnn]) = VAL(right_array[0]tpnn]); 

) 
> 

else for 0=0; j< NUM^ACTIVE_MTH_PARAMS; j++) { 
prm = active_mth_params[j]; 
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if (GT{VAL(lcft_arTay[i][pnn]),MAX^VAL(left,airay[0][pnn]))) 

MAX_VAL(left_aiTay[0][pnn]) = VAL(left_aiTay[i][pnn]); 
if (LT(VAL(left_arTay[i][pnn]), MIN_VAL(left_airay[0][pnn]))) 

MIN^VAL(left_arTay[0](pnn]) = VAL(left_array[i][pnn]); 
if (GT(VAL(iight^aiTay[i][pnn]). MAX^VAL(right^arTay(0][pnn]))) 

MAX_VAL(right_^array[0][pnn]) = VAL(right_array[i][prni]); 
if (LT(VAL(right_aiTay[i][pnn]), MIN_VAL(right_aiTay[0][prm]))) 

MIN_VAL(right_array[0][pnn]) = VAL(right_aiTay[i][prm]); 

} 

} 

) 

void calcProjection(float lcft_j)ts[][2], float right J3ts[][2]) 

{ 

register int j; 

float X, y, osc= sc, oinc = VAL(left_parameter[POLAR_INC]). 

otwist = VAL(left_parameter[POLAR_.TWIST]), 

oazim = VAL(leftj)arameter[POLAR_AZIM]); 
Matrix M; 

setPolarIiic(90,0); 

setPolarTwist(O.O); 

setPolarAziniuth(O.O); 

sc = 0.35; /* Usually it is around this number... */ 

recompute_face(); 

prcpareMatrix(M ) ; 

for 0=1; j <=np; j++) { 

world2scrcen(M. Ieftj3t[j][0]. left_pt[j][l], leftj>t[j][2), &x, &y); 

left__pts[j][0] = /•floor*/(x); 

left j)ts[j][ 1 ] = /•floorV(y); 

worId2screcn(M, right j5t[j][0]. rightj}t|j](l], right_pt[j][2], &x, &y); 
right j>tstj][0] = /•floor*y(x); 
right_ptstj][l] = /•floor*/(y); 

) 
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sc = osc; 
setPolarlnc(oinc); 
setPolarTwist(otwist); 
setPolarAzimuth(oazini); 



void calcAvg (float IeA_pts[][2], 
float right j)ts[ll2], 
float *mid_x, 
float *mid_y, 
indcx_st ♦pts_index) 

{ 

float x=0., y=0.; 
int i; 

for (i=0; i< NUM_MTH^PTS; i++) { 
if (pts_indcx{i].side = LEFT_PARAMS) { 
X Iefl:_pts[pts_index[i].pnt](0]; 
y += Ieft_pis[ptsjndex[i].pnt][l]; 

} 

else { 

X += right_pts[pts_index[i].pnt][0]; 
y += right_pts[pts_index[i].piit][l); 

} 

} 

•mid_x = X / NUM_MTH_PTS; 
♦mid_y - y / NUM^MTH.PTS; 



void setMinMax (int ♦anay) 
{ 
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int nf = array [01» i J, k. pnt, first = l.prm; 
char temp[MAX_NAME]; 
indcx^st ♦indices; 
float p[2], range; 

for (i=l; i <= nf; { 
sprintfl[tcmp,"%s.%d.pnn",out_base_name, arTay[i)); 

/♦ printfl["Checking %s\nMcmp);V 

if (!restoreParanieters(temp,tempJefl, temp_right,TRUE)) { 

printf("Wanung: file %s doesn't exist! !!\n", ten^); 

if (i=first) first++; 

else { 

if (i=first) {/* First file sets all max and min values! */ 
for 0=0; j< NUM>CTIVE^MTH^PARAMS; { 
prni = active_mth_params[j]; 

MAX_VAL(!ef^_parametcr[prm]) = VAL(temp_Ieft[prm]); 
MIN_.VAL(lcftj)arametcr[prm]) = VAL(tcmp_Ieft[pnn]); 
MAX_VAL(right_jjanimeter[prm]) « VAL(temp_right[prm]); 
MIN_VAL(rightj)arameter[prni]) = VAL(tenip_right[pini]); 

} 

} 

else { 

for j< NUM_ACnVE_MTH^PARAMS; j-f-f ) { 
prm = active_mth_params[j]; 

if {GT(VAL(tcmp_lcft[pnn]),MAX_VAL(leftj)arameter[prm]))) 

MAX_VAL(leftj3arameter(prm]) = VAL(temp_Ieft[prm]); 
if (LT{VAL(temp^left(prai]). MIN^VAL(left_j)arametcr[pnn]))) 

MIN_VAL(leftj)arameter[pim]) = VAL(temp_!eft[prm]): 
if (GT(VAL(tcmp_right[prm]), MAX^VAL(rightj)arametcr[pnn)))) 

MAX_VAL(rightjjaramctcr[pnn]) - VAL(temp^right[pnn]); 
if (LT(VAL(tcmp_.right[pnn]), MIN_VAL(right_paranicter[pnn]))) 

MIN_VAL(rightj)arameter[pnn]) = VAL(tenip_right[pnn]); 

) 
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\ 

} 

} 

printflCTotal of %d kcyframes\n", nf); 
pnntf(**Min and Max values:\n 

for 0=0; j< NUM_ACnVE^MTH_PARAMS; j-H-) { 
pnn « active_inth_parains(j]; 

if (MAX_VAL(right_paramctcr[pnn]) = MIN_VAL(rightj)arainctcr{pnn])) { 
sprint£(teinp,"%s.0.prm",out_base_nanic); 
restorcParametcrs(tcmp,temp_lcft, tenip_^right,FALSE); 
MIN__VAL(right_parameter[prm]) = MIN^VAL(temp_right[pnn]); 
MrN_VAL(left_j>arametcr[pnn]) = MIN_VAL{temp^left[prm]); 
MAX_VAL(right_j)arameter[pnn]) = MAX^VAL(temp_right[pnn]); 
MAX^VAL(leftj)arameter[pnTi]) = MAX_VAL(tempJcft[pnn]); 

} 

range = (MAX_VAL(leftj3aramctcr[prm))-MIN_VAL(lcft_j>aramctcr(prin]))*range_Iimit; 
MIN_VAL(left_parametcr[pnn]) -= range; 
MAX_VAL(left_parameter[pnn]) +~ range; 
range=<MAX_VAUrightj)arametcrtprnri])-Mm_VAL(rightj)aram 
MIN_VAL(right_parameter[pmi]) -= range; 
MAX_VAL(right_paramcter[pnn]) += range; 

printft"(%d) %s: left min = %f, left max %f\n\t\t right min %f, right max = %An", 
pnn, 

leftj3aramcter[pnn].box_label, 

MIN_^VAL(lcft_parametcr[prm]), 

MAX_VAL(lcft_parameter[prm]), 

MIN_VAL(right_parametcr[prm]), 

MAX_VAL(right_parametcr[prm])); 

} 

} 

* 

int intInAiiay(int val, int *anay) 

{ 



SUBSTITUTE SHEET (RULE 26) 



wo 97/15926 

int i, n aiTay[0]; 



73 



PCT/IB96/01056 



for i<=n; 
if (val ~ airay[i]) return (TRUE); 

return (FALSE); 

) 



float calcBeard (float p288[2]) 
{ 

float pl6[2], distance,sb; 
Matrix M; 



prcpareMatrix(M) ; 

worId2screen(M,len j)t[ 1 6] [0], left j)t[ 1 6][ I ],left_j)t[ 1 6][2],&p 1 6[0], &p 1 6( 1 ]); 
world2scrccn(M,left_pt[288][0],left j)t[288][l],leftj)t{288][2],&p288[0], &p288[l]); 
distance = dist (pi6,p288); 

) 

* Module: gimg.h - interface for gimg.c - image mils 

* By: Ranco 

* History: 

* 13.4.94: creation. 

* 2nd version (MOVIE based): 24.4.95 

^include <stdio.h> 
#include <sidlib.h> 
^include <unistd.h> 
#inciude <string$.h> 
#include <fcntl.h> 
^include <getopt.h> 
^include <gl/gl.h> 
^include <gl/device.h> 
#include <gl/image.h> 
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/* defines for image file manipulation ♦/ 



#defme GCMAPl 
Mefme GRGB3 
#defme GRGBA4 



typedef unsigned long MPIXEL; 
typedef unsigned char PELEMENT; 



#defme REDM OxOOOOOOff 
#defme GRNM OxOOOOffDO 
#defme BLUM OxOOflDOOO 
#defme ALPM OxflDOOOOO 
#defme red(x) ((x)&REDM) 
#defme grecn(x) (((x)&GRNM)»8) 
#defme blue(x) (((x)&BLUM)»16) 
#define alpha(x) (((x)&ALPM)»24) 



typedef struct { 

unsigned short xsiz, ysiz» zsi2; 
char *buf; 

unsigned short type; 
} GimgRcc, *Gimg; 



#define ImgXsi2e(i) 
#defme ImgYsize(i) 
#deftne ImgZsize(i) 
#denne ImgRast(i) 
#define ImgType(i) 



((i)->xsiz) 

((i)->ysiz) 

((i)->2siz) 

((i)->buO 

((i)->type) 



/* num colomns ♦ 
/* num rows */ 
/* # color bands •/ 



#defme ImgPixRGBadd(i, x. y) ((i)->buf + ((y)MmgXsize(i)+(x)) ♦si2eof( MPIXEL)) 
#defme ImgPixCMAPadd(i, x, y) ((i)->buf+ ((y)*ImgXsize(i)+(x)) *sizeof(short)) 
#defme ImgPixRGB(i, x, y) ( ((MPIXEL ♦)(i)->buO [(y)*UngXsize(i)+(x)]) 
#defme ImgPixCMAP{i, x, y) ( ((short ♦)(i)->buf) I(y)*ImgXsize(i)+(x)]) 
#defmc ImgInRange(i, x, y) ((x>«0.0)&&(y>«=0.0)&& \ 
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(x<(double)ImgXsi2e(i))&&(y<(doublc)lmgYsize(i))) 
Gimg GimgCreate(short xsize, short ysize, int zsize); 
int GiingInit(Gimg img, short xsize, short ysize, int zsize); 
int GimgFree(Gimg img); 
int GimgDeUGimg img); 
Gimg GimgRead(char *f)\ 
int GimgWrite(Ptr f. Gimg img); 
void GimgDisp(Gimg img, short xO, short yO); 
unsigned long GimgCalcSum(Gimg img); 
void GimgClear(Gimg img); 

Gimg GimgCreateSubCopy{Gimg orig, int xO, int yO, int width, int height); 

Gimg GimgResize (Gimg image, int w, int h, int sample); 

void GimgAvg(Gimg img, long *r, long *g, long *b, int include_zcros); 

void GimgMask(Gimg img, long r, long g, long b, int xO, int yO, int width, int height); 

void GimgSetAlpha (Gimg img, long a); 

int GimgFlip(Gimg img); 

int GimgInterlace(Gimg origin, Gimg Destination); 
int GimgDeInterlace(Gimg origin, Gimg Destination); 

* Module: image.c 

* By: Ranco 

* History: 

* 18.8.94: creation. 

^include "mv.h" 
#include "defs.h" 



extern long bgc; 



void putimage (void) 
{ 

MOVTE_showFnime(movie,(MVframc)0); 

) 
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* Module: io.c 

* By: Ranco 

* History: 

* 18.8.94: creation. 

* 2nd version (MOVIE based): 24.4.95 

^include "listh** 

/* Load points info irom filename, into PntList PI 

void loadPts (char •filename, PntList •PI) 
{ 

FILE *fp; 

int i, dummy, count = 0; 
int X, y, res; 

char namc[MAX^NAME]; 
Point P; 



if ((fp = fopcn (filename, V)) = NULL) { 
printf("%s\n",filename); 
PRINT_ERR("Can't find data filc\n"); 

} 

while(!fcof(fp)) { 
if ((res = fscanf^fp, "%d %d %d", &dummy, &x, &y)) != 3 || 
(dummy != counts 1)) { 
if (res = -1) break; 

PRINT_ERR("Wrong points file fomiatfW); 

} 

count++; 

} 

♦PI = createPntListO; 
fscek(fp,0,O); 
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for (i = 0; i < count; i++) { 
if (fscanfl[fp. "%d %d %d". Adummy, &x, &y) != 3 || 
(dummy != i+l)) { 
PRINT_ERR("Wrong points file fonnat!\n"); 

} 

P.x = {double)x; 
P.y = (doublc)y; 
addPoint(*Pl, &P); 

} 

fclose(fjp); 



* Module: list.h - interface for listx 

* By: Ranco 

* History: 

* 16.2.94: creation. 

#include <stdUb.h> 
^include <stdio.h> 
#include <niath.h> 
#include "defs.h" 

^dcfme NUM^OF^PNTS(pl) ((pl)->Size) 

^define POnsrr(pl,p) ((pl)->Points[p]) 
#derinc NUM^OF^LINES(ll) ((ll)->Si2e) 

#define LINE(il,p) ((ll)->Lincs[p]) 

#defme LENG(l) ((l).len) 

#define SOURCE(l) ((l).Pl) 

^define DEST(1) ((1).P2) 

#dcfme XCOR(p) ((p).x) 

#defme YCOR(p) ((p).y) 

/* Some point opjsrations */ 

#defme SIZE(p) (XCOR(p)*XCOR(p)+YCOR(p)*YCOR(p)) 
#defme SctPoint(p,a,b) {(p).x = (a); (p).y = (b); } 
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#define GetPoint(p,r) {(r).x=(p).x; (r).y=(p).y; } 

#dcfme AddPoints(p,pl.p2) {(p).x = (pl).x+(p2).x; (p).y = (pl).y+(p2).y; } 
Mcfinc SubPoints(p,pl,p2) {(p),x = (pl).x.(p2).x; (p).y = (pl).y-(p2),y; } 
#dcfmc MulPoint(p.c) {(p).x *= (float) (c); (p).y (float) (c); } 
#dcfiiie DivPoint(p,c) {(p).x /= (float) (c); (p).y /= (float) (c); } 
#define DotProduct(pl,p2) ((pl).x*(p2),x+(pl).y*(p2).y) 

#dcfinc SetLinc(l,pl,p2) {(l).Pl.x = (pl)->x; (l).Pl.y = (pl)->y; \ 
(l).P2.x - (p2)->x; (l).P2.y - (p2).>y; } 

typedef struct { 

float X, y; 
) Point; 

typedef stmct { 

int Size; 

Point *Points; 
} PntList^st; 

typedef PntList^st ♦PntList; 
typedef stmct { 

Point PI, P2; 

int len; 
} Line; 

typedef struct { 

int Size; 

Line *Lines; 

} LineList_st; 
typedef LineList_st ♦LineList; 

/* Prototypes ♦/ 

PntList createPntList (void) ; 

void frecPntList (PntList 1) ; 

PntList addPoint (PntList 1. Point *p) ; 

LineList createLineList (void) ; 

void freeLineList (LineList 1) ; 

LineList addLine (LineList 1, Point *P1, Point *P2); 

/* V 
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/* File: mv.h 



/* 



*/ 



/♦ Include file for the interface to movie (mv) library 



*/ 



/* 



/♦ Written by: Ranco 
/* Creation: Wed, 5/4/95 



♦/ 



/* 



♦/ 



#ifndef _MV_H 
#define _MV^H 

^include <dmedia/ci.h> 
#inciude <dmcdia/cl_cosmo.h> 
^include <movie.h> 
#include "gimg.h" 
#include "defs.h" 
^include <audionie.h> 

#defme MOVIE^ERR 0 
#dcfine MOVIE_OK I 

typedef struct { 
char file_namc[MAX_NAME]; 
MVid movie, imagetrack, audiotrack; 
int width, height; /* Size of a movie frame */ 
in! interlace; 

MVframe first, last; /* First and last frame's indices, for playback */ 

MVframc num^frames; 

MVfirame currcnt_fTame_no; 

double frame_rate; 

double audio_rate; 

int audio_width; 

char compression[20]; 

CLhandle compressor, decompressor; 

int cosmo_available; 

int panunBufllO]; 
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int n_params; 
Gimg cuirent^frame; 

int de1_img; /* a flag that indicate if current frame should be freed at the end */ 
int orientation; 

int quality; /• Jpeg Quality (0 - 100) ♦/ 
} Movie__st; 

typedef Movie_st *Movie; 



/♦•♦•*♦'♦♦* MACROS 
#define MOVIE_movieNanie(niv) 
#define MOVIE_inovieTrack(inv) 
^define MOVIE_imageTrack(n[iv) 
#definc MOVIE_audiotrack(mv) 
#d€fine MOVIE_franieWidth(mv) 
#dcfine MOVIE_franjeHcight(mv) 
#define MOVIE_numO£Fnimcs(niv) 
^define MOVIE_frameRatc(mv) 
^define MOVIE_audioRatc(niv) 
#dcfme MOVlE_audioWidth(niv) 
#define MOVIE_rirstFrame(niv) 
#define MOVIE JastFramc(mv) 



(niv->file_namc) 

(mv->movie) 

(niv->imagetrack) 
(inv->audiotrack) 

(niv->width) 

(mv->height) 

(mvGetTrackLength(niv->imagctrack)) 
(mv->frame_ratc) 
(niv->audio_rate) 

(mv->audio_width) 
{niv->first) 



(niv->last) 

^define MOVIE_getCurrentFrameNo(mv) (inv->current_frame_no) 
#define MOVIE_currcntFrame(niv) (mv->current_fTame) 

#defme MOVIE_loadCurrentFrame(mv) (MOVIE_getCurrcntFrame(niv,mv->cuiTenl_fTamc)) 
#define MOVIE_fTameOricmation(mv) (mv->orientation) 
#derine MOVI£_frameCompression(niv) (mv->conipression) 
#define MOVIE_framelnterlace(mv) (mv->interlace) 
#defme MOVIEJpegQuality(mv) (mv->quality) 



*/ 



extern Movie movie; /* Currently allow one movie in the application! 

/•♦* Prototypes ♦•♦/ 

Movie MOVrE^open (char ♦filename); 
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DMparams *MOVlE_init (Movie mv, 

char *filenaine, 

int width, 

int height, 

double framc^ratc, 

double audio_rate, 

int orientation, 

char ^con^iression, 

int interlace); 
Movie MOVIE^create (char ^filename, 
int width, 
int height, 
double fTame_rate. 
double audio_rate, 
int orientation, 
char * compression, 
int interlace, 
int jpeg_quality); 

int MOVIE^addFrame (Movie mv, MVfirame index, Gimg frame, int size); 

int MOVIE_copyFTames (Movie from^mv. Movie to_mv, MVfiame from_index, MVframe 

to^index, MVfirame count); 

int MOVIE^deleteFrames (Movie mv, MVframe index, MVframe count); 

int MOVIE_getFrame (Movie mv, MVframe index, Gimg frame); 

int MOVIE_setCunrentFrameNo(Movie mv, MVframe index); 

int MOVIE^getCunentFrame (Movie mv, Gimg frame); 

int MOVIE addCurrentFrame (Movie mv. Gimg frame, int size); 

int MOVIE_nextFrame (Movie mv); 

int MOVIE_prcvFrame (Movie mv); 

int MOVIE showFrame (Movie mv, MVframe frame_no); 

int MOVIE_bind (); 

int MOVIE_play (Movie mv); 

int MOVIE_stop (Movie mv); 

int MOVIE_home (Movie mv); 

int MOVIE_end(Movic mv); 

int MOVIE_sctStartFrame(Movie mv, MVframe frame^no); 
int MOVIE_sctEndFrame(Movie mv, MVframe frame_no); 



SUBSTITUTE SHEET (RULE 26) 



wo 97/15926 PCT/IB96/01056 

82 

int MOVIE_close (Movie mv); 

size_t MOVIE^gctFrameSizc (Movie mv, int frame no); 

Gimg MOVIE_resize (Gimg image, int w, int h, int sample); 

int MOVIE^addAudioFrame (AFfilehandle audioFile, Movie to_mv); 

AFfilehandle MOVIE_createAudioTrack (char *audio_file_name, Movie to_mv); 

#endif 

♦ Module: texture.c 

* By: Ranco 

♦ History: 

* 18.8.94: creation. 

#include "texture.h*' 
#include "x_inc.h" 
#include "general.h" 
#include "niain.h" 
#include "van.h" 
^include "mv.h" 

float texprops[] = { 

TX_MINfFILTER, TX^POINT, 

TX^MAGFILTER, TX_POINT, 
/♦ TX^MINFILTER, TX^BILINEAR, 

TX^MAGFILTER, TX^BILINEAR,*/ 

TX__WRAP, TX^CLAMP, TX^^hnjLL}; 
float tevprops[] = {TV DECAL, TV_NULL>; 

extern int texture_niap; 

void textureMapping (int op) 
{ 

long xs,ys; 
Gimg sub_img; 
int width,height; 
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if(op«=TEX_.ON) { 
texture_iiiap = TRUE; 
wire = FALSE; 
if (Ishow coord) { 
MOVJE^loadCurrentFrame (movie); 

) 

calcTextiireCoords(left_tex,right^tex); 

width ~ (int)(right_modcl-lcft_model); 

height = (int)(top_model-bottom_model); 

s u b _ i m g = 
GimgCreateSubCopy(MOVIE_cumntFrame(movie),(int)left_model,(int)bonom_model,wi^ 
height); 

zclear(); 

if (whichjmage =^ SIDE_IMG) { 
xs = ImgXsize(side_inig); 
ys = ImgYsizc(side Jmg); 

texdef2d(l, 4, xs, ys, (ujong *)InigRast(sidcJnig), 0, texprops); 

> 

else { 

GimgSetAlpha(sub_img,2551); 

texdef2d(l, 4, width,height, (u_Iong ♦)IingRast(subJmg), 0, texprops); 

} 

tevdcf(l, 0, tevpiops); 
tcvbind(TV_ENVO, 1); 
texbind(TX_TEXTURE_0, 1); 
GiingDel(sub_iing); 

} 

else { 

texture_map = FALSE; 
wire « TRUE; 
tevbind(TV_ENVO. 0); 
texbind(TX_TEXTURE_0,0); 
RefreshO; 
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} 

RedrawNeeded = TRUE; 

} 

void calcTcxturcCoords(float leftjcx[][2]. Qoat right Jex[][2]) 
{ 

register int j; 

float xs, ys,l=0.0,b=O.O; 

Matrix M; 

setModelRegionO; I* Set global ieft/right/top/bottom.model to the 
current modePs region */ 

if (which Jmage == SIDE_IMG) { 
xs = (floatKlmgXsize(side_inig)); 
ys = (float)(ImgYsize(sidc_inig)); 

) 

else { 

xs = right_inodel-lef%_model-l; /* IMPORTANT!: we assume *_model is a rectangle around 

*/ 

ys = top_model>bottom_modeM; /• the current projection of the model */ 
1 = left_model; 
b = bottom_model; 

} 

recompute_face(); 
prepareMathx(M); 
forO=l; j <=np; j++) { 

world2screen(M, left_j)t[j][0], left_j)t[j][l]. leftj)t[j][2], 
&leftjex[j][0], &lcft_tex[j]Ill); 

lcft^tex[j][0] = (lcfljcx[j][0]-l)/xs; 

lcft^tex[j][I] = (leftjex[j][ll-b)/ys; 

PRINTF5(-%d: l,b=%.lf,%.lf tex-%f, %fai"j,l,b,leftjexD][0],lefljex[j][l]); 
world2screen(M, right_pt[j][0]. rightj)t[j][I], right j)ttj][2), 
&rightjextj][0], &rightjcx[j][l]); 
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rightjcx[j][0] = (right_tex[j][0]-l)/xs; 
right_tex|j](l] = (right_tex[j][l]-b)/ys; 

} 

) 
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CLAIMS 

1. A method for automated con^juterized audio visual 
dubbing of movies comprising of the steps: 

(a) selecting from the movie a frame having a 
picture, preferably frontal, of the actor's head 
and, if available, a frame with its side profile; 

(b) marking on the face several significant 
feature points and measuring their locations in the 
frame; 

(c) fitting a generic three-dimensional head 
model to the actor's two-dimensional head picture 
by adapting the data of the significant feature 
points, as measured in stage (b), to their location 
in the model; 

(d) tracking of the said fitted three- 
dimensional head model parameters throughout the 
movie, from one frame to its successor, iteratively 
in an automated computerized way and creating a 
library of reference similarity frames ; 

(e) taking a movie of a dubber wherein the 
dubber speaks the target text; 
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(f) repeating stages (a), (b), (c), and (d) 
with the dubber; 

(g) normalizing the dubber *s mininnim and 
maximum values of each parameter to the actor's 
minimum and maximum values for the same parameters; 

(h) mapping, on a frame to frame basis, the 
two-dimensional actors face onto its three- 
dimensional head model by using a texture mapping 
technique, making use of reference similarity 
frames; 

(i) changing the texture mapped three- 
dimensional model obtained in stage (h) by 
replacing, on a frame to frame basis, the original 
mouth parameters with the mouth parameters as 
computed in stage (d) for the dubber and obtaining 
the parametric description for the new picture, 
with identical values to the original, except that 
the actor's mouth status resembles the mouth status 
of the dubber; 

(j) texture mapping the lips area of the same 
actor from a frame in the movie, with identical or 
very similar mouth status to the desired nev/ mouth 
status, onto the lips area of the actor's head 
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model for the current frame and then projecting the 
lips area from the actor's head model onto the 
current new frame (an optional stage). 

2. A method according to claim i wherein the 
parameters for controlling the three-dimensional 
head model are the position, orientation, and 
expression of the head model mouth. 

3. A method according to claim i wherein the 
significant feature points on the face marked on 
stage b are the eye corners, the mouth corners, 
and the top and bottom of the face. 

I. A method according to claim i wherein about is 
significant feature points on the face are used in 
the tracking stage. 



5. A method according to claim i wherein the movie to 
be audio visually dubbed is a sequence of one or 
more still photographs which are identically 
duplicated, frame after frame, to create a portion 
of a movie. 



. A method according to claim 5 for audio visual 
dubbing of still photographs in TV programs such as 
news from field correspondents. 
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I. h iQethod according to claim 5 wherein the actor 
does not speak such as a baby or a mute person. 

8. A method according to claim l wherein the original 
movie is an animated cartoon. 

9. A method according to claim 8 wherein the actor is 
an animal or any other non-human or non-living 
object. 

10. A method according to claim l wherein the movies 
are advertisment movies. 

II. A method according to claim l wherein the pictures 
of the movie and of the still photos are in 
electronic digital form. 

12. A method according to claim 1 wherein the pictures 
are translated into digital form, manipulated in 
digital form, and returned back to any desired 
form. 

13. A method according to claims 8 wherein the 
production of an annimated cartoon is assisted by 
drawing a straight line segment for the actor's 
mouth, drawing a small actor's picture dictionary 



wo 97/15926 



PCT/IB96/01056 



90 



containing representive reference similarity 
frajnes with completely dra%m mouths , and then 
allowing these lip-line segments to be replaced 
with the coresponding lip shapes of the dubber as 
are to be found in the actor's picture dictionary. 

14. A method according to claim l wherein the "movie 
is a painting, a drawing, or a picture. 

15 ♦ A method according to claim i for creating new 
movies or new portions of movies from a library of 
reference frames. 

16. A method according to claim 1 for the conversion 
of background narrative, or any spoken text, to an 
audio visual form. 

17. A method according to claim l wherein the dubber 
speaks the target text in either another language 
or the same language and the movie of the dubber is 
taken while the dubber performs a routine dubbing 
adaptation of the original into target text. 

18. A method for creation of a library of reference 
similarity frames in the method as defined in claim 
1. 
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19. A JDethod according to claim l wherein the 
automated computerised audio visual dubbing of 
movies is done by the software, as shown in 
Appendix 1, or by similar software, 

20. A software for use in the method as defined in 
claim 1. 

21. A movie prepared by the automated audio visual 
dubbing method as defined in the preceeding claims. 

22. A movie according to claim 22 wherein the movie is 
a video movie. 

23. A method substantially as herein before described 
and illustrated. 
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Figure I 
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Figure 2a 
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Figure 3 
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Figure 4a 




Figure 4b 
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